Methods and vehicles for capturing emotion of a human driver and customizing vehicle response

ABSTRACT

Methods and systems for determining an emotion of a human driver of a vehicle and using the emotion for generating a vehicle response, is provided. One example method includes processing captured voice data from the human driver over a period of time while the human driver operates the vehicle. The method includes analyzing the voice data to assist in prediction of the emotion of the human driver. The method includes generating the vehicle response. The vehicle response is selected in part based on the emotion that was predicted for the human driver.

CLAIM OF PRIORITY

This application claims priority as a continuation from U.S. applicationSer. No. 16/732,069, filed on Dec. 31, 2019, entitled “Methods andVehicles for Capturing Emotion of a Human Driver and Customizing VehicleResponse, which is a continuation from U.S. application Ser. No.15/470,881, filed on Mar. 27, 2017 (U.S. Pat. No. 10,535,341, issued onJan. 14, 2020) entitled “Methods and Vehicles for using Determined Moodof a Human Driver and Moderating Vehicle Response,” which is acontinuation of U.S. application Ser. No. 15/351,422, filed on Nov. 14,2016, (U.S. Pat. No. 9,672,823, issued on Jun. 6, 2017) entitled“Methods and Vehicles for Processing Voice Input and use of Tone/Mood inVoice Input to Select Vehicle Response,” which is a continuation of U.S.application Ser. No. 14/949,883, filed on Nov. 24, 2015, (U.S. Pat. No.9,493,130, issued on Nov. 15, 2016) entitled “Methods and Systems forCommunicating Content to Connected Vehicle Users Based DetectedTone/Mood in Voice Input,” which claims priority from U.S. ProvisionalPatent Application No. 62,185,578, filed on Jun. 27, 2015, entitled“Methods and Systems for Communicating Content to Connected VehicleUsers Based Detected Tone/Mood in Voice Input,” and 62/254,858, filed onNov. 13, 2015, entitled “Methods and Systems for Communicating Contentto Connected Vehicle Users Based Detected Tone/Mood in Voice Input,”which are incorporated herein.

U.S. application Ser. No. 14/949,883, filed on Nov. 24, 2015 is acontinuation-in-part of application Ser. No. 14/275,569, filed on May12, 2014 (U.S. Pat. No. 9,467,515, issued on Oct. 11, 2016) entitled“Methods and Systems for Sending Contextual Content to ConnectedVehicles and Configurable Interaction Modes for Vehicle Interfaces,”which is herein incorporated by reference.

U.S. application Ser. No. 14/949,883, filed on Nov. 24, 2015 is acontinuation-in-part application of U.S. application Ser. No.13/784,823, filed on Mar. 5, 2013 (U.S. Pat. No. 9,285,944, issued onMar. 15, 2016) entitled “Methods and Systems for Defining Custom VehicleUser Interface Configurations and Cloud Services for ManagingApplications for the User Interface and Learning Setting Functions,”which claims priority to U.S. Provisional Patent Application No.61/745,729, filed on Dec. 24, 2012, and entitled “Methods and SystemsFor Electric Vehicle (EV) Charging, Charging Systems, InternetApplications and User Notifications,” and which are herein incorporatedby reference.

U.S. application Ser. No. 14/949,883, filed on Nov. 24, 2015 is also acontinuation-in-part of U.S. application Ser. No. 13/842,158, filed onMar. 15, 2013 (U.S. Pat. No. 9,229,905, issued on Jan. 5, 2016) andentitled “Methods and Systems for Defining Vehicle User Profiles andManaging User Profiles via Cloud Systems and Applying Learned Settingsto User Profiles,” which is herein incorporated by reference.

U.S. application Ser. No. 13/784,823, filed on Mar. 5, 2013 is acontinuation-in-part of U.S. application Ser. No. 13/452,882, filed Apr.22, 2012 (U.S. Pat. No. 9,123,035, issued on Sep. 1, 2015) and entitled“Electric Vehicle (EV) Range Extending Charge Systems, DistributedNetworks Of Charge Kiosks, And Charge Locating Mobile Apps,” whichclaims priority to U.S. Provisional Application No. 61/478,436, filed onApr. 22, 2011, all of which are incorporated herein by reference.

FIELD OF THE EMBODIMENTS

The present invention relates to systems and methods for customizingvehicle response to user/driver voice input, and methods for processingvoice input to detect tone and/or mood of user, and providing a vehicleresponse customized for the detected tone and/or mood of the user, andsystems for exchanging information with cloud-based processing systems.

BACKGROUND

Vehicles, such as motorized vehicles and electric vehicles have beenaround for some time. Vehicles provide a way that enable humans to drivefrom place to place. In today's world, vehicles have become anindispensable mode of transportation, and provide the freedom to travelat any time of day and for various distances. Vehicles can be publicallyoperated or can be privately owned. Humans most commonly operatevehicles, no matter the type, whether electric or combustion enginebased. In recent years, technology has been advancing to allow forbetter wireless interfacing and networking with vehicles.

It is in this context that embodiments of the invention arise.

SUMMARY

The methods, systems and apparatus are provided, which includeprocessing systems for executing vehicle responses to voice input. Invarious configurations, a user's tone of voice is analyzed to determinematches in predefined tones. The tones, in some embodiments, are matchedto voice profiles that determine or correlate to a selected vehicleresponse. The vehicle response to voice input can include, for example,making a setting, finding a map, finding directions, settingentertainment functions, looking up information, selecting acommunication tool, making a call, sending a message, looking up acontact, looking up a calendar event, performing an Internet search,controlling a system of the vehicle, etc.

Broadly speaking, the processing defined herein enables the vehicleresponse to be tailored to respond to the user's voice input in a waythat respects or understands the user's possible mood or possible stateof mind. For example, if the user's tone implies that the user isrushed, the system (e.g., vehicle electronics, software, cloudprocessing, and/or user connected devices) will process that tone in thevoice and will provide a vehicle response in a more expedited manner, orwithout further queries. If the tone implies that the user is relaxed,the system may provide supplemental information in addition toresponding to the voice input. For example, if the user asks for adining spot near a park, the system may also recommend nearby coffeeshops, discounts for parking, nearby valet parking, and/or promotions.However, if the user appears stressed or rushed, the supplementalinformation may be omitted and a response can be quick and to the point.In some embodiments, in addition to detecting a user's tone and/ordetecting user mood, the system can identify a geo-location context andan environmental context. These additional contextual data points can beused to provide further customized vehicle response and/orrecommendations to the user of the vehicle (i.e., driver and/orpassengers).

In one embodiment, a method for determining a mood of a human driver ofa vehicle and using the mood for generating a vehicle response isprovided. One example method includes capturing, by a camera of thevehicle, a face of the human driver. The capturing is configured tocapture a plurality of images over a period of time, and the pluralityof images are analyzed to identify a facial expression and changes inthe facial expression of the human driver over the period of time. Themethod further includes capturing, by a microphone of the vehicle, voiceinput of the human driver. The voice input is captured over the periodof time. The voice input is analyzed to identify a voice profile andchanges in the voice profile of the human driver over the period oftime. The method processes, by a processor of the vehicle, a combinationof the facial expression and the voice profile captured during theperiod of time to predict the mood of the human driver. The methodgenerates the vehicle response that is responsive to the mood of thehuman driver. The vehicle response is configured to make at least oneadjustment to a setting of the vehicle. The adjustment is selected basedon the mood of the human driver. The vehicle response can be used tomake the driver more calm and/or assist in reducing distracted driving.

In one embodiment, a method for processing voice inputs at a vehicle isprovided. The method includes determining a profile for a user of thevehicle using electronics of the vehicle. The profile is part of a useraccount and the vehicle is provided with wireless communicationcircuitry for accessing a server over a network for exchanginginformation regarding the vehicle, use of the vehicle and use of theprofile. The method includes receiving via a microphone of the vehicle avoice command from the user of the vehicle. The electronics of thevehicle processing the voice command to enable a two-way conversationexchange between the user and the vehicle. The method acts to access, byelectronics of the vehicle, data for learned behavior of the user. Thelearned behavior is associated to the profile for the user. The methodprocesses the voice command to identify a type command. The type ofcommand is one of an instruction to make a change to a settingassociated with the vehicle or a request for interfacing with a remoteservice over the network. The method includes identifying a vehicleresponse for implementing the type of command identified for the voicecommand and processing, by the electronics of the vehicle, the vehicleresponse. The vehicle response is moderated based at least in part onthe learned behavior of the user.

In one embodiment, a vehicle having an on-board computer for processingvoice input is provided. The vehicle has a microphone interfaced withthe on-board computer and memory for storing a sample of audio datareceived from the microphone. The audio data is a voice input directedto the vehicle. A processor of the on-board computer is configured toprocess the sample of audio data to identify markers in frequency and/ormagnitude. The markers are used to define an audio signature for thevoice input, and the audio signature is used to identify a voiceprofile. The voice profile is used to identify a vehicle response forthe voice input, and the voice profile is associated with tone of voiceused in the voice input. The vehicle response acts to direct a vehiclesystem function to take an action based on the voice input and thevehicle response is adjusted based on the tone of voice. The tones ofvoice are associated to inferred moods of the user which include one ormore of a normal mood, a frustrated mood, an agitated mood, an upsetmood, a hurried mood, an urgency mood, a rushed mood, a stressed mood, acalm mood, a passive mood, a sleepy mood, a happy mood, or an excitedmood, or combinations of two or more thereof. The action to be taken isbased on the voice input and is one of a command to input a setting ofthe vehicle, a command requesting information, a request to access data,a request to communicate, or a combination thereof.

In one embodiment, a method for processing voice inputs at a vehicle isprovided. The method includes sending, to a cloud processing server,data from the vehicle. The vehicle includes an on-board computer forprocessing instructions for the vehicle and processing wirelesscommunication to exchange data with the cloud processing server. Themethod enables receiving, at the vehicle, data for a user account to usethe vehicle. The cloud processing server uses the user account toidentify a user profile of a user. The method further enables receiving,from the cloud processing server, voice profiles for the user profile.Each voice profile is associated with a tone identifier. The voiceprofiles for the user are learned from a plurality of voice inputs madeto the vehicle by the user in one or more prior sessions of use of thevehicle. The method receives, by on-board computer, a voice input, forthe voice input and processes, by on-board computer, the voice input toidentify a voice profile for the voice input. The vehicle response isthen generated for the voice input. The vehicle response is selectedbased on the tone identifier of the identified voice profile.

In some embodiments, the tone identifiers identify a mood of the user.In some embodiments, the mood of the user includes one or more of anormal mood, a frustrated mood, an agitated mood, an upset mood, ahurried mood, an urgency mood, a rushed mood, a stressed mood, a calmmood, a passive mood, a sleepy mood, a happy mood, an excited mood, orcombinations of two or more thereof.

In some embodiments, the tone identifiers identify a dialect of theuser.

In one embodiment, a tone identifier is descriptive or representative ofan actual tone of voice used by the user when making a voice input to avehicle's voice control interface, which includes at least onemicrophone used to capture the user's voice. The microphone may beintegrated into the vehicle, e.g., near the steering wheel, dash, visor,a seat, etc., and can be connected to electronics of the vehicle. In oneembodiment, the voice input is processed to capture an audio sample ofthe voice input. The audio sample may be the entire command, part of thecommand, or multiple commands, statements, one or more spoken words,verbal sounds, verbal gestures, grunts, moans, yells, expletives, and/orcourteous statements words or the like. In general, the audio samplewill include some audible sound that can be captured. The audio sample,in this example, refers to an amount of audio to cache or save toperform the analysis.

Once captured, the analysis can include frequency sampling. Forinstance, in one embodiment, the voice input can be in the transmittedvoice frequency of about 300-3400 Hz, and the sampling frequency isabout 8 kHz. In some embodiments, the transmitted bandwidth frequencycan be in the range of 50-7000 Hz, and the sampling frequency can beabout 16 kHz, and in other embodiments, the transmitted bandwidthfrequency can be in the range of 20-20,000 Hz, and the samplingfrequency can be in the range of about 44.1 kHz. For most voice inputs,the sampling frequency is typically about 8 kHz.

Optionally, the captured audio sample can be processed to remove noise,such as ambient noise, voice noise of other passengers, music playing inthe vehicle, tapping noises, road noise, wind noise, etc. The audiosample is then processed to produce an audio signature. The audiosignature may be in the form of an analog signal or digital code. Theaudio signature may identify certain frequencies in the spoken words,audio modulations, frequency peaks, peak-to-peak identifiable patterns,spikes, pauses, or other characteristics that can identify ordistinguish when one spoken word, e.g., command, is said to have aparticular tone. In some embodiments, in addition to voice input, othersensors can detect the magnitude of sensed touch inputs, physiologicalcharacterizes of the user's body, motions, demeaned, and combinationsthereof.

By way of example, when a user says, “find me a hotel”, that statementcan be said in a normal voice, e.g., if the driver is not tired. If thedriver is tired or sleepy, the user may voice these words at a slowerpace, or with less emphasis on different words. The tone of voice isthus viewed as a tone identifier. The tone identifier, in oneconfiguration, can be identified from the audio signature produced fromthe voice input. The audio signature can then be used to identify a typeof vehicle response that is most appropriate for the tone in which thevoice input is made. As described below, depending on the tone used toprovide the voice input, different types of vehicle response can be madefor the same grammatical input. In some cases, the vehicle response canbe expedited to avoid delay. In some cases, the vehicle response caninclude additional information, e.g., recommendations, extrainformation, advertisements, etc.

In some embodiments, the dialect of the user includes one or more of alanguage dialect, a country dialect, a geographic region dialect, an agedialect, or combinations of two or more thereof.

In some embodiments, the user account is accessed by receiving logindata at the vehicle.

In some embodiments, the login data includes one or more of a password,a wireless login code, a biometric identifier, a gesture, a pairingbetween a device of the user and logic of the on-board computer.

In some embodiments, the on-board computer is configured select thevehicle response from among a plurality of alternate vehicle responsefor the voice input and associated tone identifier.

In some embodiments, the voice profiles and associated tone identifiersare stored in storage of the cloud processing server and a copy of thevoice profiles and associated tone identifiers is stored in storage ofthe vehicle.

In another embodiment, a method to be executed by a processor of acomputer of a vehicle is provided. The method includes receiving voiceinput of a user at the vehicle to command input or request informationfrom a voice input interface of the vehicle. The method further includesanalyzing the voice input of the user to identify a tone of the voiceinput. The method then functions to identify a voice profile of the userfor the identified tone. The voice profile is used to select a vehicleresponse that is moderated for the tone of the voice input.

In some embodiments, the voice profile identifies a type of vehicleresponse that is customized for the user, based on the identified tonein the voice input by the user.

In some embodiments, a plurality of voice profiles are associated withthe user, each voice profile identifies a preferred vehicle response forthe user based on the tone.

In some embodiments, a method can include receiving the voice input,sampling the received voice input and then identifying a frequency andmagnitude in the voice sample. The frequency and magnitude identifyingmarkers in the voice input. The method includes identifying a tone thatbest matches the identified markers. The tone that best matches theidentified markers is for the user. The method also includes identifyinga voice profile for the identified tone. The voice profile is used toselect a type of vehicle response. The vehicle response includes one ofan input to the vehicle or a system input, or a response requested fromthe vehicle or setting to be set by the vehicle.

In some embodiments, the identifying the tone identifies when additionalinformation is to be provided or not provided to the user based on thetone.

In some embodiments, the additional information includes advertisingdata, or discount data, or related information that is in addition tothe requested information or instructed command found in the voiceinput.

In some embodiments, the voice profile use based on the tone assists inmoderating when to provide additional information to the user or when toavoid providing additional information to the user.

In some embodiments, the tone identifies a mood or condition of theuser.

The contextual relevance can be based on information associated with thegeo-location of the vehicle, the state/condition of the vehicle, learnedpreferences, information in user online accounts, information fromsocial networks, information from patterns of use by the user,information based on the time of day, week, month or based on whencertain inputs or data that is requested or looked up by the user, orcombinations of one or more of the foregoing. The intersection oranalysis of these information points allow for the system to providecontextually relevant information to the vehicle, when the user needsthe information and/or when it is anticipated that the information willbe consumed, used, accessed, viewed, or desired. In one embodiment,these processing acts to filter out information that is not useful orrelevant for the particular time, circumstances, state of the vehicle,geographic location, time of day, etc., and as a result, reduced orun-needed or un-desired information is provided or sent to the vehiclefor presentation, which beneficially acts to reduce driver distraction.For instance, the vehicle may determine that the user is highlydistracted based on the tone of the voice the user is providing as aninput to the vehicle. Thus the vehicle may elect to limit interactionwith the user until the user appears to be less distracted. However, thevehicle can monitor the context and environmental inputs to determinethat the user seems distracted but is in fact only having a conversationwith a passenger and may still want to interact with the vehicle with ahigh level of detail. The vehicle would be able to understand that aconversation is taking place in the vehicle and learn that the userstill wants to be interacted with even though indicators suggest theuser is not requiring more interaction. The vehicle and learn and storethis contextual environmental input for the next time the user seemsdistracted but is only having a conversation with a passenger and wouldstill like additional interaction with the vehicle.

In one embodiment, an interaction mode can define a single setting, or aplurality of settings. If a plurality of settings is defined for oneinteraction mode, individual settings of that interaction mode may bemodified by the user or automatically in view of learned patterns,learned behavior, or the like. In some embodiments, the learned patternscan be identified from a collection of similar users. For example, ifother users are registered with a cloud service for connected vehicles,typical settings, changes, control modifications, preferences,demographic preferences, regional/cultural preferences, languagepreferences, etc. can be mined to identify patterns. In one embodiment,these patterns can be mined without requiring the actual identify of auser, so that privacy protection can be maintained for all useraccounts.

In some implementations, the learning and predicting embodiments mayutilize learning and prediction algorithms that are used in machinelearning. In one embodiment, certain algorithms may look to patterns ofinput, inputs to certain user interfaces, inputs that can be identifiedto biometric patterns, inputs for neural network processing, inputs formachine learning (e.g., identifying relationships between inputs, andfiltering based on geo-location and/or vehicle state, in real-time),logic for identifying or recommending a result or a next input, a nextscreen, a suggested input, suggested data that would be relevant for aparticular time, geo-location, state of a vehicle, and/or combinationsthereof. In one embodiment, use of machine learning enables the vehicleto learn what is needed by the user, at a particular time, in view ofone or more operating/status state of the vehicle, in view of one ormore state of one or more sensors of the vehicle. Thus, one or moreinputs or data presented to the user may be provided without explicitinput, request or programming by a user at that time. Overtime, machinelearning can be used to reinforce learned behavior, which can provideweighting to certain inputs.

In one implementation, the at least one aspect of one of the preferencesis data obtained from an internet service, wherein the internet serviceis one of a website, or a calendar, or social network website, or a newssite, or a dictionary site, or mapping service, or a to-do list, or aphone list, or a merchant website, or a shopping website, or a couponsite, or a discount site, or gasoline price site, or an electric vehicle(EV) charge locator service, or an EV charge reservation service, or ane-payments site, or an energy pricing site, or a route mapping service,or a traffic service or site, or a movie site, or a music site, ortravel site, or a vehicle site, or vehicle manufacturer site, or arental car site, or an airline reservation site, or a restaurant findingsite, or a review site, or a weather site, or a loyalty rewards site, ora database, or a historical driving database, or a vehicle-to-vehicledatabase, or a holiday calendar, or the internet.

In one embodiment, a method is provided. The method includes receivinggeographic locations of a vehicle over time, at a server configured toexecute cloud services for a user account. The user account identifyingthe vehicle and the user account further includes a profile for a userof the vehicle. The method also includes accessing the profile toidentify a history of use of the vehicle for the user. The method alsoincludes generating, from time to time, a plurality of learnedpreferences that are associated to the profile of the user by examiningthe history of use of the vehicle for the user. The history of use ofthe vehicle includes geographic locations of the vehicle, inputs made toa user interface of the vehicle at particular times or when the vehicleis at particular geographic locations. The method also includes, for acurrent geographic location and for a current time, identifyingsupplemental content for display on the user interface. The supplementalcontent is contextually related and filtered based on the currentgeographic location of the vehicle, the current time, and the learnedpreferences. Then, sending the supplemental content to the vehicle overa wireless network. The supplemental content is configured for displayon the user interface of the vehicle; the method is executed by aprocessor.

In one implementation, the user interface of the vehicle is integratedin one or more display screens of the vehicle, the one or more displayscreens being configured to display a level of information items basedon an interaction mode. In one implementation, settings of theinteraction mode define a style of user interfaces of the displayscreens of the vehicle, the style of user interfaces of the displayscreen identify one or more of text format, text size, icon types,simplicity of interface features, types of gauges, clutter levels,skins, wallpaper, styles, designs, colors, and/or voice input/outputdisplay features. In one implementation, some of the supplementalcontent includes identification of goods or services proximate to atleast one of the geographic locations of the vehicle, the goods orservices identified being filtered to include goods or servicespredicted to be of interest for display based on examination of theprofile of the user account, and the profile of the user account furtherincludes data regarding user preferences and context informationregarding likelihood of interest for goods or services at other timessimilar to a current time.

In some embodiments, the vehicle display of the vehicle includes any oneor more of a main dashboard display, or a center console display, or acombined main dashboard and center console display, or a glass surface,or a windshield display, or a window display, or a touch surfacedisplay, or a headrest display, or a movable display, or a wirelessdisplay, or a wire-connected display, or combinations of one or morethereof.

In one embodiment, the custom configuration is generated using tools andprograms made available on a website. The tools and programs may beexecuted by computers, such as computers of a data center to providecloud based processing. The data centers can be distributedgeographically and the communication to specific vehicles can bedynamically assigned to various geographic data centers, as the vehiclesmove around geographically.

In some embodiments, the mood of the user could be used to determinethrough algorithmic, ambient, stored and learned conditions, a state ofintoxication.

In some embodiments, the vehicle systems can be allowed to listen to invehicle conversations between a driver and a passenger or betweenpassengers. The vehicle parses the main data points in the conversationto determine an appropriate response. For example, if a passenger in thefront seat is expressing that he or she is hot; the vehicle could askthe passenger if a lower temperature is desired. If the system detectsthat a passenger in the rear seat is mentioning that they would likemore air conditioning, the vehicle can either ask the driver to grantadditional AC to the rear of the vehicle or automatically send more ACto the rear of the vehicle. The vehicle responses can be tailored to theuser by not only learning but also by allowing the user to set the levelof response.

In some embodiments, the vehicle system can be set by the user tofunction with varying levels of autonomy. If the user of the vehiclewould like the vehicle to ask for less permissions to change vehiclesettings and just perform the actions the vehicle seems fit, a user canset the vehicle to a high level of autonomy. Conversely, if a user wantsthe vehicle system to ask more confirmations and more permissions, thevehicle system can be set by the user to run with less autonomy. Varyinglevels of autonomy between very minimum autonomy and maximum antimonycan be set by the user.

In some embodiments, the vehicle can learn over time the level ofautonomy it should operate under based on learning in past decisionpoints. For instance, the user can tell the vehicle system by voice ortouch that it is not necessary to ask for permission to lower thevehicle temperature when it is deemed too hot for the user. After a dataset has been compiled where the system attains a level of certainty, thesystem will no longer ask for permission to lower the temperaturebecause it has learned that, for instance, in 4 of the last 4 times thesystem asked to lower the temperature, the user told the system not toask for permission. Consequently, the 5^(th) time the system determinesit is time to lower the temperature in the vehicle it will not ask forpermission. Alternatively, the vehicle may become too autonomous for auser's liking, so the user can ask the system to ask for permission nexttime it sets the seat warmer for example. This aids the vehicle systemin determining the environmental inputs present when the user was okwith autonomy and when the user was not only with autonomy. For example,the vehicle system will incorporate into learning history and algorithmsfor future decision making that the user likes not asking for permissionto set the seat heater when the ambient temperature in the vehicle isbelow 45 degrees Fahrenheit. However, the user does not like the vehicleto automatically set the seat heater without asking when the ambienttemperature in the vehicle is above 45 degrees.

The methods, systems and apparatus are provided, which includeprocessing systems for executing vehicle responses to touch input. Invarious configurations, a user's touch characteristic is analyzed todetermine matches in predefined touch characteristics. The touchcharacteristic, in some embodiments, is matched to touch profiles thatdetermine or correlate to a selected vehicle response. The vehicleresponse to touch input can include, for example, making a setting,finding a map, finding directions, setting entertainment functions,looking up information, selecting a communication tool, making a call,sending a message, looking up a contact, looking up a calendar event,performing an Internet search, controlling a system of the vehicle, etc.

Broadly speaking, the processing defined herein enables the vehicleresponse to be tailored to respond to the user's touch input in a waythat respects or understands the user's possible mood or possible stateof mind. For example, if the user's touch characteristic implies thatthe user is rushed, the system (e.g., vehicle electronics, software,cloud processing, and/or user connected devices) will process that touchcharacteristic in the touch and will provide a vehicle response in amore expedited manner, or without further queries. If the touchcharacteristic implies that the user is relaxed, the system may providesupplemental information in addition to responding to the touch input.For example, if the user elects to look for a dining spot near a parkvia the graphical interface using a touch input, the system may alsorecommend nearby coffee shops, discounts for parking, nearby valetparking, and/or promotions if the system determines that the touchcharacteristic indicates a relaxed or normal user mood. However, if thesystem determines that the user appears stressed or rushed based on thetouch characteristic the user's touch, the supplemental information maybe omitted and a response can be quick and to the point. In someembodiments, in addition to detecting a user's touch characteristicand/or detecting user mood, the system can identify a geo-locationcontext and an environmental context. These additional contextual datapoints can be used to provide further customized vehicle response and/orrecommendations to the user of the vehicle (i.e., driver and/orpassengers).

In one embodiment, a method for processing touch inputs at a vehicle isprovided. The method includes sending, to a cloud processing server,data from the vehicle. The vehicle includes an on-board computer forprocessing instructions for the vehicle and processing wirelesscommunication to exchange data with the cloud processing server. Themethod enables receiving, at the vehicle, data for a user account to usethe vehicle. The cloud processing server uses the user account toidentify a user profile of a user. The method further enables receiving,from the cloud processing server, touch profiles for the user profile.Each touch profile is associated with a touch characteristic identifier.The touch profiles for the user are learned from a plurality of touchinputs made to the vehicle by the user in one or more prior sessions ofuse of the vehicle. The method receives, by on-board computer, a touchinput, for the touch input and processes, by on-board computer, thetouch input to identify a touch profile for the touch input. The vehicleresponse is then generated for the touch input. The vehicle response isselected based on the touch characteristic identifier of the identifiedtouch profile.

In some embodiments, the touch characteristic identifiers identify amood of the user.

In some embodiments, the mood of the user includes one or more of anormal mood, a frustrated mood, an agitated mood, an upset mood, ahurried mood, an urgency mood, a rushed mood, a stressed mood, a calmmood, a passive mood, a sleepy mood, a happy mood, an excited mood, orcombinations of two or more thereof.

In some embodiments, the touch characteristic identifiers identify aphysical profile of the user. For example, a user may touch a graphicaluser interface and the vehicle learns the size of the user's touch printand in doing so stores the touch characteristic. This touchcharacteristic may indicate that the user has large fingers with largesurface area and this tells the vehicle not to misinterpret a touch bythe user as a user touching hard, but simply the size of the touch printis relative to the size of the user's finger.

In some embodiments, the accuracy of a user's touch characteristic canbe measured. For instance, a graphical user interface may provide abutton for the user to touch. The vehicle can measure using on boardelectronics and screen mapping coordinates or other known touch screencoordinate tracking methods, how close to the center of the button theuser's touch was registered. Varying degrees of accuracy can be thenrecorded after every touch to determine how consistently accurate orinaccurate the user is being during a session of driving in terms oftouching the button. The vehicle can compare the touch characteristicsof the user in terms of accuracy with known touch characteristicprofiles stored on the vehicle or in the cloud to determine how accuratethe user is being. For instance, if the user is consistently 5coordinate points or less away from the center of the button during acurrent driving session, and stored touch characteristic profilessuggest that 5 coordinate points or less is quite accurate, the vehiclewill determine that the user is being quite accurate. This data ishelpful for identifying how focused or distracted the user is. In thelast example, the user is quite accurate which may suggest that thedriver is not rushed, is relaxed and does not have a lot ofdistractions. Additionally, this may aid in determining the state ormood of the user to the converse. If the user is determined to beconsistently quite inaccurate, this may indicate that he user is rushed,agitated, tired, sleepy, or distracted. Based on the determination thevehicle has made using touch, touch characteristics and comparing touchprofiles to known touch profiles, an inference can be made on the partof the vehicle on how to respond. If the user is rushed, a clutter freeGUI may appear. If a user is sleepy, a brighter LCD screen may be set oralerts may sound.

In some embodiments the duration of a user's touch characteristic can bemeasured. For instance, a graphical user interface may provide a buttonfor the user to touch. The vehicle can measure using on boardelectronics, screen capacitive sensors touch screen touch durationcapturing methods, how long the user's finger was touching the button onthe screen. Varying degrees of duration can be then recorded after everytouch to determine touch duration patterns during a session of driving.The vehicle can compare the touch characteristics of the user in termsof touch duration with known touch characteristic profiles on thevehicle computer or stored in the cloud to determine the averageduration of a user's touch and what the duration of touch could mean.For instance, if the user is consistently registering GUI touch inputsthat last less than one second during a current driving session, andstored touch characteristic profiles suggest that touches lasting onaverage less than one second is very brief, the vehicle will determinethat the user is being quite brief with their touch inputs. This data ishelpful for identifying how focused or distracted the user is. In thelast example, the user is touching input screens with a high level ofbrevity which may suggest that the driver is rushed, is not relaxed andmay be distracted. Additionally, this may aid in determining the stateor mood of the user to the converse. If the user is determined to beconsistently making touch inputs that have a duration of an average ofmore than 1 second, this may indicate that he user is not rushed, isrelaxed, could be tired, or sleepy. Based on the determination thevehicle has made using touch, touch characteristics and comparing touchprofiles to known touch profiles, an inference can be made on the partof the vehicle on how to respond. If the user is rushed, a clutter freeGUI may appear. If a user is sleepy, a brighter LCD screen may be set oralerts may sound.

In some embodiments the intensity of a user's touch characteristic canbe measured. For instance, a graphical user interface may provide abutton for the user to touch. The vehicle can measure using on boardelectronics, screen capacitive sensors touch screen touch intensitycapturing methods, how much pressure the user's finger was exerting onthe button on the screen. Varying degrees of intensity can be thenrecorded after every touch to determine touch intensity patterns duringa session of driving. The vehicle can compare the touch characteristicsof the user in terms of touch duration with known touch characteristicprofiles on the vehicle computer or stored in the cloud to determine theaverage touch intensity pressure of a user's touch and what the pressurereading of a touch could mean. For instance, if the user is consistentlyregistering GUI touch inputs that are light in pressure during a currentdriving session, and stored touch characteristic profiles suggest thattouches with light pressure are below average in touch pressure, thevehicle will determine that the user is being lighter than usual orlighter than normal with their touch inputs. This data is helpful foridentifying how focused or distracted the user is. In the last example,the user is touching input screens with a low level of pressure whichmay suggest that the driver is not rushed, is relaxed and may not bedistracted. Additionally, this may aid in determining the state or moodof the user to the converse. If the user is determined to beconsistently making touch inputs that have a high degree of pressureintensity, this may indicate that he user is rushed, is angry, or couldbe distracted. Based on the determination the vehicle has made usingtouch, touch characteristics and comparing touch profiles to known touchprofiles, an inference can be made on the part of the vehicle on how torespond. If the user is rushed, a clutter free GUI may appear. If a useris sleepy based on very light pressure, a brighter LCD screen may be setor alerts may sound.

In some embodiments the position of a user's touch characteristic can bemeasured. For instance, a steering wheel may be equipped with electronicand sensors that can register not only where the steering wheel is beinggrasped but also by how many hands. The vehicle can measure using onboard electronics, screen capacitive sensors touch screen touchintensity capturing methods, how defensively or offensively the user isgrasping the stealing wheel, stick shift knob and other graspable areasin a vehicle such as an arm rest. Varying positions can be then recordedafter every touch to determine touch position patterns during a sessionof driving. The vehicle can compare the touch characteristics of theuser in terms of touch position with known touch characteristic profileson the vehicle computer or stored in the cloud to determine the normalposition of hands on a steering wheel for instance and what the positionreading of a touch could mean. For instance, if the user is consistentlygrasping the steering while at a “10 and 4” position current drivingsession, and stored touch characteristic profiles suggest that a “10 and4” steering wheel grasp means the user is using two hands, the vehiclewill determine that the user is being more cautious than usual or morecautious than normal with their steering while grasp position. Touchintensity and touch duration may further help to identify how hard theuser is grasping the steering while and how long an intense steeringwhile grasp at “10 and 4” as transpired.

In one embodiment, this data is helpful for identifying how focused ordistracted the user is. In the last example, the user is touching inputpositions suggesting the driver is being cautious and may be trying tofocus. Thus based on the determination the vehicle has made using touch,touch characteristics and comparing touch profiles to known touchprofiles, an inference can be made on the part of the vehicle on how torespond. In this example, the vehicle has determined it is not a goodtime to distract the user so a clutter free GUI may appear and lessvehicle to user interaction may transpire. In another example, the touchcharacteristics of the user may indicate that the position of the user'shands on the steering while show that the user is using consistentlyusing one hand to steer the vehicle and with light pressure. The vehiclemay determine that the user is relaxed and may be in need of moreinteraction with the vehicle.

In some embodiments, the vehicle may user only one touch characteristicto make an inference on how to tailor responses to inputs based on touchcharacteristics and the comparison with touch characteristic profiles onthe vehicle computer or stored in the cloud

In some embodiments a combination of touch characteristics such as touchaccuracy, touch duration and or touch intensity may be computed by thevehicle system to make in inference on how to tailor responses to theuser based on touch characteristics.

In some embodiments, the vehicle may user only one voice tone to make aninference on how to tailor responses to inputs based on tone and thecomparison with tone profiles on the vehicle computer or stored in thecloud.

In some embodiments a combination of voice tones may be computed by thevehicle system to make in inference on how to tailor responses to theuser based on tone.

In some embodiments a combination of touch characteristics such as touchaccuracy, touch duration and or touch intensity may be used inconjunction with a combination of voice tones computed by the vehiclesystem to make in inference on how to tailor responses to the user basedon tone and touch characteristics.

In some embodiments, the on-board computer is configured select thevehicle response from among a plurality of alternate vehicle responsefor the touch input and associated touch characteristic identifier.

In some embodiments, the touch profiles and associated touchcharacteristic identifiers are stored in storage of the cloud processingserver and a copy of the touch profiles and associated touchcharacteristic identifiers is stored in storage of the vehicle.

In another embodiment, a method to be executed by a processor of acomputer of a vehicle is provided. The method includes receiving touchinput of a user at the vehicle to command input or request informationfrom a touch input interface of the vehicle. The method further includesanalyzing the touch input of the user to identify a touch characteristicof the touch input. The method then functions to identify a touchprofile of the user for the identified touch profile. The touch profileis used to select a vehicle response that is moderated for the touchcharacteristic of the touch input.

In some embodiments, the touch profile identifies a type of vehicleresponse that is customized for the user, based on the identified touchcharacteristic in the touch input by the user.

In some embodiments, a plurality of touch profiles are associated withthe user, each touch profile identifies a preferred vehicle response forthe user based on the touch characteristic.

In some embodiments, a method can include receiving the touch input,sampling the received touch input and then identifying accuracy,duration, pressure intensity, or position in the touch sample. Theaccuracy, duration, pressure intensity, or position identifying markersin the touch input. The method includes identifying a touchcharacteristic that best matches the identified markers. The touchcharacteristic that best matches the identified markers is for the user.The method also includes identifying a touch profile for the identifiedtouch characteristic. The touch profile is used to select a type ofvehicle response. The vehicle response includes one of an input to thevehicle or a system input, or a response requested from the vehicle orsetting to be set by the vehicle.

In some embodiments, the identifying the touch characteristic identifieswhen additional information is to be provided or not provided to theuser based on the touch characteristic.

In some embodiments, the additional information includes advertisingdata, or discount data, or related information that is in addition tothe requested information or instructed command found in the touchinput.

In some embodiments, the touch profile use based on the touchcharacteristic assists in moderating when to provide additionalinformation to the user or when to avoid providing additionalinformation to the user, or when to interact with the user or when toavoid interaction with the user

In some embodiments, the touch characteristic identifies a mood orcondition of the user.

In one embodiment the user of a vehicle may elect to determine orpre-program how the vehicle should respond based on user defined voiceor touch inputs. The user may elect to draw a connection manuallybetween certain tones and or touch characteristics and manuallydetermined vehicle responses.

In one embodiment, a vehicle may be set by a user to only automaticallyrespond based on learning to a user's voice or touch inputs only, onlyrespond based on user defined vehicle responses to user voice or touchinputs, or respond in a combination of learned and user definedpreferences dictating how a vehicle should respond based on tone ortouch characteristics.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates an example of a user interfacing with a vehicle, viavoice input, in one embodiment.

FIG. 2 illustrate a flow diagram of processing performed (e.g., by thevehicle), to identify a tone of voice and/or other user behavior todetermine the vehicle response, which will be applied by the vehicleelectronics, in one embodiment.

FIG. 3 illustrates a system and use diagram, of the configuration of theuser interface, in accordance with one embodiments of the presentinvention.

FIG. 4 illustrates an example of user interfacing with a display in thedashboard of vehicle, in one embodiment.

FIG. 5 illustrates how the custom configuration that provides theinterface defined by the user is downloaded to the vehicle electronicsand the display of the vehicle in one embodiment.

FIG. 6 illustrates an example of synchronizing other applications to avehicle display, which may be customized by the user, for oneembodiment.

FIG. 7 illustrates example where the user holding a portable device cansynchronize data from the portable device directly with the display ofthe vehicle, in one embodiment.

FIG. 8 describes a system in which a user interacts with a model viewcontroller software environment useful for processing APPS using APIs onvehicles with vehicle operating systems capable of processing computercode, in accordance with one embodiment.

FIG. 9A describes how vehicle on board computer with input out/putsystem useful for accepting input, processing input and displayingresults in conjunction with stored computer readable programs orfunctions in the forms of APPs may be structured, in accordance with oneembodiment.

FIG. 9B describes one example of how stored data and functiondeclarations may be compiled to provide intermediary access to avehicle's computer controlling vehicle systems, in accordance with oneembodiment.

FIG. 9C describes a set of computer readable and executable code thatcan be compiled together by a third party APP developer in the form ofan APP, in accordance with one embodiment.

FIG. 10 describes the stepped flow of events as a user interacts with anAPP, in this case, an HVAC APP, in accordance with one embodiment.

FIG. 11 Describes further ways an APP may take, process and produceresults, in accordance with one embodiment.

FIG. 12 shows assumptions and reasoning logic useful in providinginformation to a user or vehicle in response to multiple points of data,history of use, preferences, and combinations thereof, in accordancewith one embodiment.

FIG. 13 illustrates an example of a vehicle dashboard having a reducedclutter display, contextual recommendations and notifications providedto the vehicle computer from cloud services, in accordance with oneembodiment.

FIGS. 14A-15 illustrate examples of contextual information analysis forinformation that concerns operational states of the vehicle, personalinformation associated with the user, learned patterns or behaviors ofthe user, the time of day, geo-location, and other dynamically changingparameters that are synthesized to identify contextual recommendationsthat are displayed to the user's interfaces and automatic launching ofapplications and content in applications based on information that isbelieved to be needed at the time and at the location and for the userand based on other factors, in accordance with one embodiment.

FIGS. 16A-16C illustrate examples of interaction modes selected for thevehicle, and the manner in which content items are displayed on the oneor more displays, the manner in which information is surfaced orprovided to the driver or occupants, and the dynamic automaticpresentation of content on the displays which are contextually relatedto factors that include the geo-location, content in a user's calendar,learned preferences or behaviors, the time of day, and other parametersin accordance with one embodiment.

FIG. 17 illustrates an example of various interaction modes, and amethod for selecting the interaction modes on a display of a device, thevehicle, a mobile device, or any other Internet connected device, inaccordance with one embodiment.

FIG. 18 illustrates one example flow of operations used to providecontextually relevant information to the displays of the vehicle, forone embodiment.

FIGS. 19-21 illustrate example flow diagrams for various embodiments,which allow for communication of intelligently selected recommendations,information, surfacing of content, and selection of interaction modes orsettings for interaction modes, in accordance with several embodiments.

FIG. 22 illustrates an example of cloud services being able to interactwith multiple Internet services, cloud data structures, cloud databases,third-party websites, information services, user vehicles, and otherinformation and data that can be accessed for intelligentlycommunicating supplemental content to the user account for display onone or more displays of a vehicle, and for the dynamic receipt of userinteractions over time from the connected vehicle, in accordance withseveral embodiments.

FIG. 23 illustrates an example system diagram of a vehicle interfacingwith cloud services and receiving vehicle input or recommendations basedon processed information obtained from a data store that defines a userdata graph 108′, said user data graph 108′ is compiled or grows overtime based on information collected, learned, input, obtained andprocessed for the user over time, in accordance with some embodiments.

FIG. 24 illustrates an example of a vehicle interfacing with a user thatprovides voice input, and the vehicle (or portable device incommunication with the vehicle computer) exchanges data with cloudprocessing, the tone of the user's voice input is used to determine orselect a type vehicle response, in accordance with some embodiments.

FIG. 25 illustrates a plurality of different user voice inputs, whichmay be provided by the user or users over time, and each voice input isexamined for tone, which identifies the state of the user which is usedto provide a vehicle input, in accordance with some embodiments.

FIG. 26 illustrates a table that illustrates examination of tones ofvoice input and identification a voice profile, in accordance with someembodiments.

FIG. 27 illustrates processing of voice profiles based on input tones ofthe user's voice and the resulting vehicle response, in accordance withsome embodiments.

FIG. 28 illustrates an example of a user providing voice input to avehicle, the vehicle being in communication with cloud processing, andthe vehicle using real-time data obtained from cloud processing or datacached at the vehicle in order to identify a vehicle response for thetone and associated voice profile (e.g., the response being moderatedfor the tone detected in the user input), in accordance with oneembodiment.

FIG. 29 illustrates an example flow diagram of receiving voice input andidentifying a voice profile of the user for the identified tone, whichis used to select a vehicle response that is moderated for the tone ofthe voice input provided by the user, in accordance with someembodiments.

FIG. 30 illustrates one example of a vehicle driving compartment and thepassenger seat, wherein a plurality of physical controls can becustomized by the use of dynamically changeable screens, which allowphysical controls the change functionality, display complexity, orinformation types, for blending the physical controls with aninteraction mode of a vehicle, in accordance with several embodiments.

FIG. 31 illustrates a hybrid physical/display controls with anassociated interaction mode customization.

FIG. 32 illustrates an example of a car dashboard having interactionstyles and screens and settings therefore, in accordance with oneembodiment.

FIG. 33 shows an example of a vehicle that includes vehicle processingfor handling voice input of a user, the voice input is captured by oneor more microphones in the vehicle and processing enables identificationof an audio signature that represents the user's tone in his or hervoice, to then enable selection of a specific vehicle response type,among various options, in accordance with some embodiments.

FIG. 34 illustrates processing that can be performed to analyze touchinput to determine a vehicle response, in accordance with oneembodiment.

FIG. 35 illustrates an example similar to FIG. 23 , except that theseinputs relate to touch inputs. In this example, user touch analysis canbe performed in 3502 in order to identify the type of touch, inaccordance with one embodiment.

FIG. 36 illustrates an example of using touch input to identify a user'smood and then to identify a vehicle response, in accordance with oneembodiment.

FIG. 37 illustrates several examples of using touch characteristics (TC)in order to determine the mood, state, or mental state of the user whiledriving, in accordance with one embodiment.

FIG. 38 illustrates an example of processing operations that can be usedto map touch characteristics to touch profiles.

FIG. 39 illustrates an example of various touch inputs that identifytouch characteristics of happy, urgent, and sleepy.

FIG. 40 illustrates an example of a vehicle performing analysis of touchinput 4010, and providing a vehicle response.

FIG. 41 illustrates an example of method operations that can beperformed in order to utilize touch input to identify a type of vehicleresponse to be performed by the vehicle.

FIG. 42 illustrates an example of vehicle processing that can beperformed to identify passive ambient conversational inputs, inaccordance with one embodiment.

FIG. 43 illustrates an example of using conversation keywords (CKW) toidentify the types of vehicle responses that will be provided to theuser.

DETAILED EMBODIMENTS

Embodiments of the present invention define methods, systems andapparatus for use in vehicles. The methods, systems and apparatusinclude electronics of vehicles that drive display devices in vehicles.In some embodiments, computer systems and displays are integrated withthe vehicle and in other embodiments, vehicle electronics communicatewith portable devices, and in other embodiments, portable devices andthe electronics of the vehicle work together to exchange information,user interface data, data for applications, data for displays, etc. Insome embodiments, internet connections provided by the vehicle are used,and in other embodiments, internet connections provided by a portabledevice is used, and in still other embodiments, communication to theinternet can be via the vehicle communication systems and also those ofa portable device. The vehicle, in one embodiment, is referred to as aconnected vehicle, as the vehicle uses data and communicationinformation from a remote server or service or cloud system, etc. In thevarious embodiments described in this document, it should be understoodthat embodiments may be combined to define specific implementations, andin some cases, implementations can be defined by combining only specificelements described herein.

In each of the implementations, the systems provide physical technicalresults. These technical results are not results that can be manuallyproduce by paper and pencil, but instead require processing by clientdevices, servers, and distributed internet systems, so as to provideintelligent data that can be used in an efficient manner via theconnected vehicle. Furthermore, the processing operations describedherein provide useful solutions the problems associated with distracteddriving. Distracted driving occurs most often when users are required toprovide input to a vehicle while driving. The vehicle inputs can includetouch inputs, voice inputs, gesture inputs, and any other type of inputthat requires the user to remove his or her primary concentration fromdriving. Therefore, improving the recognition of voice input andallowing the vehicle to provide a vehicle response that is mostappropriate to the mood, mental state, and/or desires of the user whiledriving will improve the ability of the driver to continue toconcentrate on driving. The less the user is required to focus onproviding the correct input to the vehicle, the more the driver is ableto concentrate on driving and avoiding potential disastrous accidents.These technical operations are performed by specific processing circuitsof the vehicle, enter specialized to communicate withapplication-specific interfaces to control functions of the vehicle.These control functions can include navigation functions, entertainmentfunctions, safety functions, operational functions, and communicationfunctions. These operations, when returned to the driver as vehicleresponses should be as desired by the user, to avoid having the user tore-program or read provide instruction to the vehicle. Accordingly, theanalysis of voice, analysis of touch, analysis of gestures, analysis offacial recognition, and other combined biometric analysis will allow thevehicle to provide the most optimal vehicle response, and input.

The methods, systems and apparatus are provided, which includeprocessing systems for executing vehicle responses to voice input. Invarious configurations, a user's tone of voice is analyzed to determinematches in predefined tones. The tones, in some embodiments, are matchedto voice profiles that determine or correlate to a selected vehicleresponse. The vehicle response to voice input can include, for example,making a setting, finding a map, finding directions, settingentertainment functions, looking up information, selecting acommunication tool, making a call, sending a message, looking up acontact, looking up a calendar event, performing an Internet search,controlling a system of the vehicle, etc. In general, the vehicleresponse is tailored to respond to the user's voice input in a way thatrespects or understands the user's possible mood or possible state ofmind. For example, if the user's tone implies that the user is rushed,the system (e.g., vehicle electronics, software, cloud processing,and/or user connected devices) will process that tone in the voice andwill provide a vehicle response in a more expedited manner, or withoutfurther queries. If the tone implies that the user is relaxed, thesystem may provide supplemental information in addition to responding tothe voice input. For example, if the user asks for a dining spot near apark, the system may also recommend nearby coffee shops, discounts forparking, nearby valet parking, and/or promotions. However, if the userappears stressed or rushed, the supplemental information may be omittedand a response can be quick and to the point. For example, the responsecan be to show five restaurants near the park, and associatedcontact/map info, reservations links, or the like. For the relaxedinquiry, the system may attempt to refine the request and as, what typeof food are you interested in, or identify coupons available for certainnearby restaurants, before providing a list of four restaurants near thepark, and associated contact/map info, reservations links, or the like.

In some embodiments, in addition to detecting a user's tone and/ordetecting user mood, the system can identify a geo-location context andan environmental context. These additional contextual data points can beused to provide further customized vehicle response and/orrecommendations to the user of the vehicle (i.e., driver and/orpassengers).

One example method for processing voice inputs can include sending, to acloud processing server, data from the vehicle. The vehicle includes anon-board computer for processing instructions for the vehicle andprocessing wireless communication to exchange data with the cloudprocessing server. The method enables receiving, at the vehicle, datafor a user account to use the vehicle. The cloud processing server usesthe user account to identify a user profile of a user. The methodfurther enables receiving, from the cloud processing server, voiceprofiles for the user profile. Each voice profile is associated with atone identifier. The voice profiles for the user are learned from aplurality of voice inputs made to the vehicle by the user in one or moreprior sessions of use of the vehicle. The method receives, by on-boardcomputer, a voice input, for the voice input and processes, by on-boardcomputer, the voice input to identify a voice profile for the voiceinput. The vehicle response is then generated for the voice input. Thevehicle response is selected based on the tone identifier of theidentified voice profile.

In some embodiments, the tone identifiers identify a mood of the user,and the mood of the user can include, without limitation, one or more ofa normal mood, a frustrated mood, an agitated mood, an upset mood, ahurried mood, an urgency mood, a rushed mood, a stressed mood, a calmmood, a passive mood, a sleepy mood, a happy mood, an excited mood, orcombinations of two or more thereof.

In some embodiments, the tone identifiers identify a dialect of theuser, and the dialect of the user includes one or more of a languagedialect, a country dialect, a geographic region dialect, an age dialect,or combinations of two or more thereof.

The contextual relevance can be based on information associated with thegeo-location of the vehicle, the state/condition of the vehicle, learnedpreferences, information in user online accounts, information fromsocial networks, information from patterns of use by the user,information based on the time of day, week, month or based on whencertain inputs or data is requested or looked up by the user. Theintersection of these information points allows for the system (e.g.,server, vehicle computer, user device, or combinations thereof) toprovide contextually relevant information to the vehicle, when the userneeds the information, so that distracted driving can be reduced.

Further, the vehicle can be customized or the user account/profile canbe customized for vehicles to allow interaction modes to be used.Interaction modes define the way of access-input, look and feel,content, simplicity, complexity, skins, etc. of the user interfaces orcontrols of the vehicle. By enabling this level of customization,vehicles can be configured or customized over time to the way each useris most comfortable, thus reducing distracted driving. Thiscustomization can also extend to physical inputs, such as knobs,switches, buttons, dials, etc. The customization can be, in oneembodiment, by adding display screens to physical inputs to definehybrid inputs. The display screens can be on the physical inputs orbeside the inputs, so that the content displayed can change, thuschanging the functionality of each or some or one of the physical inputbased on an interaction mode or setting. By providing this level ofcustomization, distracted driving can be reduced, as the vehicle iscustomized to what the user is most comfortable with and can thusconcentrate on driving.

The Internet services provide access to cloud services. The cloudservices provide access to user accounts and access to settings,configurations, applications and other customization defined by theuser. Customization can include user interface customization of avehicle display or displays. The customization can include the abilityto select specific applications (APPS) to be activated by the vehicleand interfaced via the display or displays, voice input, touch input,etc. The customization is also provided with a learning engine thatlearns use by the user, and automatically implements settings orprogramming to aspects of the user interface. The programming caninclude automatic programming at certain times, days, months, years,etc., and can be updated or molded over time as the user continues touse the vehicle UI.

The user's saved UI configuration may also be transferred to the displayof the rented (or other vehicle) vehicle. A best-fit configuration canalso be generated using the user's profile selections, so that theconfiguration provided for the other vehicle will closely resemble orappear as it does for the configured vehicle. In other embodiments, theuser's use metrics can be monitored. The use metrics can include use ofAPPS, use be of system components of the vehicle, use of the vehicle,environment conditions, and historical actions taken by the user via theinput/output controls of the vehicle (e.g., buttons, levers, keys, fobs,display selections, display interface actions, communication actions,etc.).

These historical actions can then be used to define learned actions. Thelearned actions can be analyzed to change configuration settings in theuser's saved profile. For instance, if the user uses a particular APPevery day at a particular time, that APP icon can be surfaced to thedisplay or preset to start. The APP can then provide information to theuser at about the same time the user normally needs the information.Other historical use patterns can be monitored and such data can besaved to the user's profile. The data can then be used by algorithmsthat build assumptions based on historical inputs by a user as well asenvironmental inputs, location inputs, vehicle diagnostic inputs;internet connected marketing deals, the user's calendar, trafficconditions as well as news. The assumptions the algorithm builds arethen processed into decisions and actions by an additional algorithmicprocess to activate local or remote audio and visual alerts, changevehicle systems, display information on a vehicle's displays and requesta decision from a user locally or remotely to complete an action.

A number of embodiments are described below, with reference to specificimplementations that refer to vehicles, but such implementations shouldbe broadly construed to include any type of vehicle, structure orobject. Without limitation, vehicles can include any type of movingobject that can be steered, and can include vehicles that are for humanoccupancy or not. Vehicles can include those that are privately owned,owned by corporations, commercially operated vehicles, such as buses,automobiles, trucks, cars, buses, trains, trolleys, etc. Examplevehicles can include those that are combustion engine based, electricengine (EV) based, hybrids, or other types of energy source vehicles.

A cloud processing system, as described herein, will include systemsthat are operated and connected to the Internet or to each other usinglocal networking communication protocols. A cloud processing system canbe defined as interconnected and distributed physical or virtualsoftware defined network that utilizes virtual or physical processingand storage machines that enable various applications and operatingsystems to facilitate the communication with and between various clientdevices (vehicles, user devices, structures, objects etc.). Thecommunication with and between the various client devices will enablethe cloud processing system to deliver additional processinginformation, data, and real-time metrics concerning data obtained fromother processing systems as well as client feedback data. Thedistributed nature of the cloud processing system will enable users ofvarious vehicles, structures and objects to access the Internet, and bepresented with more flexible processing power that will provide therequested services in a more effective manner.

The processing systems can be defined from various data centers thatinclude multiple computing systems that provide the processing power toexecute one or more computer readable programs. The processing of thecomputer readable programs can produce operations that can respond torequests made by other processing systems that may be local to avehicle's electronic system. For example, a vehicle can includeelectronics that utilize memory and a processor to execute programinstructions to provide services.

In other embodiments, the electronics of a vehicle can synchronize witha user's portable electronics. The user's electronics can include, forexample mobile devices that include smartphones, tablet computers,laptop computers, general-purpose computers, special purpose computers,etc. The various computing devices of the vehicle, and or the computingdevices of the user (smart devices) can be connected to the Internet orto each other. Provided that a user has access or account access to thecloud service, the cloud processing services on the Internet can provideadditional processing information to the electronics of the vehicle.

The wireless communication can include cellular tower communication thatcouples and communicates through various networks to the Internet, toprovide access to cloud processing 120. Other methods can includeproviding Wi-Fi communication to local Wi-Fi transmitters and receivers,which communicate with cloud processing 120. Other types ofcommunication can include radio frequency communication, such as802.11.ac, 802.11ad and subsequent wireless networking protocols,Bluetooth communication or combinations of Wi-Fi and Bluetooth. Itshould be understood that vehicle electronics 161 can communicate withcloud processing 120 via any number of communication methods, so long asexchanges of data can be made with cloud processing 120 from time totime.

The communication can be made by vehicle electronics 161 while thevehicle is on or when the vehicle is off, so long as communication andprocessing circuitry of vehicle electronics 161 has a power source. Thepower source can include battery power that powers vehicle electronicsto communicate with cloud processing 120 when vehicle is turned off.When vehicle is turned on, the battery that drives vehicle electronics161 can be recharged.

FIG. 1 illustrates an example of a user 121 interfacing with a vehicle160, via voice input 140. In this example, the voice input is providedto electronics of the vehicle to provide a command, setting, requestdata, request information, make changes, adjustments or communicate withothers or the vehicle. In one embodiment, the vehicle is incommunication with cloud processing 120. Cloud processing 120 includescloud services. In one embodiment, a database 112 is managed by cloudservices. User accounts and profiles can be saved in the database 112.In some embodiments, cloud services includes user voice analysis module108 for tone identification. In one embodiment, voice analysis module108 may use voice stress analysis (VSA) to record psychophysiologicalstress responses that are present in the human voice when a personsuffers psychological stress in response to a stimulus (e.g., aquestion), and the consequences of the person's answer may be dire. Inone embodiment, recorded “micro tremors” in a person's voice areconverted via the algorithm into a scorable voice gram, e.g., audiosignature. Overtime, the system learns a user's voice and micro-tremors,inflections, emotional sounds and/or repeating patterns. In someembodiments, a person's gender can also be used to adjust vehicleresponse to sensed input. In still other embodiments, the cloudprocessing system, as it manages user accounts for many users, can tapinto the learning of individual users to find similarities to improvemappings for vehicle responses to detected user moods, stress,physiological condition, heart rate, person's body heat, eye gazepatterns, etc.

In one embodiment, a pitch detection algorithm (PDA) may be used toestimate the pitch or fundamental frequency of a quasiperiodic orvirtually periodic signal, usually a digital recording of speech or amusical note or tone. This can be done in the time domain or thefrequency domain or both the two domains. A PDA may be used in variouscontexts (e.g. phonetics, intensity of command, stress identification,and/or mood detection) and so there may be different demands placed uponthe algorithm. In one embodiment, a variety of algorithms may be used toidentify different data determining factors in a person's producedpitches in voice output.

Detecting emotional information can also use passive sensors whichcapture data about the user's physical state or behavior withoutinterpreting the input. The data gathered is analogous to the cueshumans use to perceive emotions in others. For example, a video cameramight capture facial expressions, body posture and gestures, while amicrophone might capture speech. Other sensors can detect emotional cuesby directly measuring physiological data, such as skin temperature andgalvanic resistance. In some embodiments, a camera or IR camera candetect temperature changes in a person's skin. For instance, if a useris stressed, the blood rushing to a person's face may elevate the heatpattern or sensed heat from that person's face.

Recognizing emotional information requires the extraction of meaningfulpatterns from the gathered data. This can be done using machine learningtechniques that process different modalities, such as speechrecognition, speech waveforms, natural language processing, or facialexpression detection, and produce either labels (i.e. “sad,” “mad,”“happy,” “hurried,” “stressed,” etc.).

In the embodiments described herein, mood and emotions can be used tomodify the response provided by a vehicle, e.g., in response to a voicedriven input. In one embodiment, sensors can be used to detect changesin the autonomic nervous system that are exposed by a user's speech. Forinstance, the way a user alters his or her speech can be used asinformation to produce systems capable of recognizing affect based onextracted features of speech. For example, speech produced in a state offear, anger or joy becomes faster, louder, precisely enunciated with ahigher and wider pitch range. Other emotions such as tiredness, boredomor sadness, lead to slower, lower-pitched and slurred speech. In oneembodiment, emotional speech processing recognizes the user's emotionalstate by analyzing speech patterns. Vocal parameters and prosodyfeatures such as pitch variables and speech rate may be analyzed throughpattern recognition, e.g., using one or more microphones of a vehicle.As noted, in addition to detecting sound/voice of a user, one or morecameras may also detect facial patterns, which may be used to detectmood or reinforce a determination that a particular mood is present inthe driver or occupant of the vehicle.

In still other embodiments, one or more cameras internal to the vehiclemay be used for facial detection, gesture detection, breach of security,intruder detection, motion detection, light detection, color detection,moving object detection, and motions of objects and passengers in thevehicle. In one configuration, cameras may be embedded in a dash boardof a vehicle, on a display screen of a vehicle, on a roof liner, on aseat, on a glove box, on a touch panel, or in several of these places.In one embodiment, stereo cameras may be used to detect a volume ormonitor a volume of space in the vehicle. In one configuration, thevolume of space may be located where gesture detection is to take place.For example, one or more cameras may be placed so that a location nearthe center console, display screen or some user interface is monitored.In monitoring this volume of space, the cameras can capture multiplevideo frames, which can be analyzed to determine what gesture the useris making with his or her hand. For instance, the user may make a swipegesture, and the direction of the swipe gesture many be monitored, andbased on processing and matching to know gestures, the swipe gesture canbe translated to a specific input command for the vehicle. In somecases, the motion and number of fingers used to make certain gesturescan be captured.

If one finger is used, the motion and direction of the tracked finger,using the cameras, can be used to determine whether a specific input isdesired. In some cases, when two fingers are used, the two fingers areidentified. In still other cases, when one or select fingers are placedin the volume of space monitored by cameras, the motions, changes infinger positions, surfaces touched, tapped, slid, contacted, etc., maybe monitored to determine what type of gesture input is desired. Instill other embodiments, the gesture detection using cameras may beperformed in combination with other tracking, sensor or detectionfunctions. For example, the gesture camera tracking/detection can beperformed in combination with voice input processing, face detection,and other biometric detection systems. By combining sensed and detectedactions by the user, wherein said actions can be sound, voice, motion,gestures, voice tones, voice level, force used when making touch inputs,swipe and input profiles, environmental air detection, gas detection,eye detection, gaze detection, retina scan sensing, etc., it is possibleto identify false positives and also optimize the vehicle response towhat the user actually intended as the input.

In some embodiments, training can be performed, wherein a user is askedto show a sad face, a mad face, a frustrated face, a surprise face, andurgent face, etc., and images of the users face can be added to adatabase as templates. These templates can be used during real-timeprocessing to perform matching against detected facial expressions madeby the user when inputting information to a vehicle. For example, if inaddition to voice analysis of the user to detect whether the user isfrustrated, image data can also be used to confirm the frustrated stateagainst saved templates. The templates can be saved in memory of thevehicle, and utilized when determining whether the user is in aparticular state of mood. The templates can identify facialcharacteristics, such as the placement of the user's mouth, the way thateyes are opened or closed, the cheekbone characteristics, and otherbiometric determining of features. Thus, facial templates of the usercan be made during operational use of the vehicle, during training, orcalibration. As such, the templates used to identify the users face canbe used in conjunction with analysis of the voice input to betteridentify a match of the user's mood.

In one configuration, a user may provide training input data tocalibrate use of the voice analysis module 108. The user may be asked toread phrases, stories, ask questions, respond to questions, or asked torespond to emotionally charged questions. In some examples, the systemmay produce a training session that askes the user to respond tomultiple questions using different intentional tones. For instance, theuser may be asked to say “open the door” in a normal mode, an excitedmode, in a mad mode, in a stress mode, etc. In a similar manner, theuser may be asked to say, “change the station,” “map address,” “gethelp,” “call home,” “find a gas station,” “find a charge station,”“sound alarm,” etc. As can be appreciated, these phrases can beintentionally exaggerated by the user when asked to do so in training,and this training can be transferred as a starting calibration for thevehicle. In one embodiment, the training and calibration can beprocessed via an online application, such as one that can identify auser account and administer the training. The training and/orcalibration data can then be transferred to the vehicle as a settingupdate, e.g., via the internet using a wireless connection. Once thevehicle is in normal use, as commands, inputs or response are providedto the vehicle, the vehicle can refine the response for the user, e.g.,the user profile.

In one embodiment, a tone-vehicle response mapping module is alsoprovided, for each user. For example, cloud processing managesinformation for registered users, and information for specific users isprocess and learned, and used to develop a user data graph 108′. Theuser data graph 108′ is formed based on user settings made by the user,behavior, historical actions, etc. A voice profile module 109 is alsoprovided. The voice profile 109 receives tone information from module106, in order to assist in selecting a voice profile, such as happy,sad, angry, frustrated, hurried, etc. Over time, the mapping of tones tovoice profiles can change, and this information is reinforced and savedto the user data graph 108′. In one embodiment, the user data graph 108′is used by vehicle electronics to make determinations as to a vehicleresponse, for the voice input. In one embodiment, the user data graph108′ may be constructed and refiled by cloud processing 120. In otherembodiments, the user data graph 108′ may be constructed and refiled byvehicle electronics. In still other embodiments, the user data graph108′ may be processed and formed by the cloud processing 120 andtransferred to the vehicle as a cached copy. In one configuration, ifthe vehicle has a cached copy, the vehicle can more quickly processvoice inputs, determine tone and apply the response based on theidentified voice profile. In some embodiments, as inputs are made at thevehicle, updates are sent to cloud processing 120 to update or refinethe user data graph 108′.

FIG. 2 illustrates an example of processing of voice input, in oneembodiment. For example, the vehicle receives voice input 130. The voiceinput is examined in operation 131 against user data graph 108′ oflearned behavior and/or tones of voice. Based on this processing, inoperation 132, a voice profile is identified for the voice input. Inoperation 133, the voice response is identified for the voice profile.As described throughout the specification, the voice profile may becustomized for the specific user, and the vehicle response applied inoperation 134 is based on the user's data graph 108′. As a result, thevehicle response conforms to the user's voice profile, so that theresponse will be received well by the user or throttled.

In some embodiments, various user states can be detected. In addition touser's tone of voice and mood, other conditions can be detected, in someembodiments.

For example, fatigue can be determined in numerous ways or combinationof ways, for example, (1) length of driving, (2) GPS destinationdistance, (3) how long has the car been on last time since the carstopped, (3) driving patterns, (4) hitting road dividers, (5) notstaying in the lines, (6) impaired speech, (7) car can give driver atest and measure accuracy, tone, slurring, seated position, (8) knownangles of the head vs the body suggesting sleeping, (9) measuringnodding off, (10) audio detection, (11) listening for snoring, (12)listening for delirious speech.

In one embodiments, if threshold percentage of conditions exist, theelectronics/systems of the vehicle can provide verbal feedback/output tothe user (e.g., via speakers of the vehicle or speakers of a portabledevice, or combinations thereof). This audio output can say, forexample, “Would you like me to open the windows for some fresh air?”,“Big Coffee down the road has coffee on sale,” “SLEEP-E hotel is in 10miles, book hotel room?”. In some embodiments, if no passenger ispresent, and driver fails to respond, sound audio noise to wake driverup, turn AC on to stimulate driver, etc.

In some embodiments, Low Fuel or battery level/range prediction can beprovided. This can be determined in numerous ways. For example, a fuelor battery measurement is taken, a calculation of range vs. destination,a calculation of range vs. next available fueling or battery/chargingstation. If a threshold percentage of conditions exist, e.g., greaterthan 50%, or greater than 80%, etc. If the threshold is met, as preset,the systems of the vehicle can provide a visual and/or auditory alert. Averbal query may be provided, e.g., “Your fuel is low, would you likedirections to your favorite “SHELL” or “CHEVRON” or the closer?”, “Yourdriving range is 60 miles, however no fueling stations or chargingstations are available after the next fueling station located 5 milesfrom your location”, “Highly advisable, stop at “kwick-E Gas” located at123 main street. Map this fueling station?”.

In some embodiments, GPS and Gyroscope data may be used to determinelevel of interaction. For example, if speed is low or car is stopped,more interaction will appear or be audible. If vehicle is on a windyroad, the car asks less questions or more simple questions. If vehicleis in heavy traffic, the vehicle moves most important GUI to the screenclosest to the windshield line or sight or projects. In one embodiment,important GUI information is moved onto the windshield.

In one embodiment, mood can be detected in various ways or combined waysas described throughout this application. For example, and withoutlimitation, mood may be automatically detected. In one embodiment, arushed mood is sensed. Rushed moods may be sensed when user is shortwith answers, user lets car to leave user alone, rate of speech isfaster than normal, distress sensed in the voice, visual confirmationuser is very focused on the road, grip intensity on wheel, user's seatedposition is upright, or combinations thereof. In one embodiment, thevehicle can react in various ways, for example, the vehicle queries areclear, the vehicle queries are brief, the GUI on all screens change tolimit distractions, routes are changed to quickest based on traffic, andnot most direct, accident information is automatically displayed, etc.

In one embodiment agitated mood may also be sensed. For example, sensingcan include, without limitation, user is short (e.g., curt), user isusing known expletives, user is yelling, user asks vehicle to stopasking questions, driving is erratic compared to normal drivingpatterns, biometric sensors, blood pressure, position of hands onsteering wheel, how hard the wheel is being grasped. In one embodiment,if user's heart rate is high, the vehicle may react in various ways.Heart rate may be sensed from user devices that are worn or from devicesof the vehicle, e.g., sensors on the steering wheel, or some vehicleservice, or optically (non-touch) sensing. In one embodiment, thevehicle may react by scaling back the number of questions posed to theuser while in the vehicle. In another embodiment, the vehicle suggeststurning off queries visually instead of verbally. In another embodiment,the vehicle GUI becomes more standard and easy to read. In still anotherembodiment, the vehicle changes ambient lighting to a calming hue.

In still other embodiments, good mood may also be sensed. For example,the user is jovial, the user's seated and wheel grasp position isrelaxed, user's heart rate is at rest, user is asking more questions ofthe vehicle, user driving very normally as compared to historicaldriving, or combinations of two or more thereof. The vehicle may reactin various ways, for example, a number of queries increase, morequestions about settings, and more convenience based settings andquestions asked, suggest coupons and asks to route to redeem, GUI offersmore metrics and feedback, etc.

In some embodiments, biometrics may be used to sense other parameters ofa user. For example, surfaces on a vehicle that the user/driver may comeinto contact may be analyzed. For instance, door surfaces may bemonitored, floor surfaces may be monitored, dashboard surfaces may bemonitored, and seat surfaces may be monitored. The surfaces may beconfigured with various sensors, e.g., biometric sensors. The sensorsmay or may not be visible or may be covered by liners, cloths, plastics,steel, metals, glass, etc. The materials, in some embodiments, areconfigured based on the sensing surface. For example, if the surface adoor panel, the surface may be covered with leather, glass, plastic,transparent materials, heat sensitive materials, and/or tactilematerials. In some embodiments, when a user is in contact with somesurface of the vehicle, e.g., the driver, or passenger or both,particular users maybe be optionally monitored for mood and/or wellness.For instance, for the driver, it would be a benefit to monitor a driverto see if the driver is getting tired, sleepy, over agitated, stressed,high energy, lower energy, or simply non-responsive. This monitoring canoccur over time, e.g., over a period of a trip or multiple trips,wherein some trips having similar patterns can be used to establishreference points. The reference points can be used to build a userprofile and learning curves, which assist a vehicle system to makebetter identification of mood and/or wellness. Using this information,which can be shared and/or co-processed in the cloud, the vehicle can beconfigured to apply responsive action. The responsive action may be, forinstance, waking up the driver with sound, air, warnings, and/orsignals. Suggesting that the driver relax. Notifying a third party thatthe driver is tired (e.g., an employer of a vehicle driver), notifyingthe driver to change the temperature, automatically change thetemperature, etc.

In one embodiment, a biometric broadly refers to metrics related tohuman characteristics. Biometrics authentication is used also usable asa form of identification and access control. In some embodiments,biometrics can also be used to identify individuals in groups that areunder surveillance. Biometric identifiers are the distinctive,measurable characteristics used to label and describe individuals.Biometric identifiers are often categorized as physiological versusbehavioral characteristics. Physiological characteristics are related tothe characteristics of the body. Examples include, but are not limitedto fingerprint, palm veins, face recognition, DNA, palm print, handgeometry, iris recognition, retina and odor/scent. Behavioralcharacteristics are related to the pattern of behavior of a person,including but not limited to typing rhythm, gait, and voice, voice tone,voice inflections, voice speed or slowness, sharp voice harmonics, orvoice identifiers/patterns. Based on the detected biometric, e.g.,voice, automatic detection can point to one or more emotions. As notedabove, various sensors can be used to detect a driver's affect,including camera, microphone, and heart rate, blood pressure and skinconductance sensors.

In further embodiments, mood may directly affect intensity of feedback.If Angry “turn DOWN the music,” then then vehicle lowers music by 10x,for example. If happy “turn down the music,” then vehicle lowers musicby 3×.

In still other embodiments, mood sensor and vehicle reactions arerefined over time with more use. For instance, if a user is angry, yetstill wants more interaction and more questions from the vehicle, thevehicle learns that the user does not mind queries and will adjust thenumber of queries based on how the user reacts over time.

In further embodiments, vehicle personality can be refined. For example,the vehicle can have personality attributes. For example, what canchange a vehicle personality automatically may vary from user to userand vehicle to vehicle, over time and based on learned behaviors. In oneembodiment, user's login or profile information may be used. Forinstance or in addition, users age, users accent, etc. In someembodiments, the vehicle can determine or predict what the user's nativelanguage is based on the type of accent, users language, users use ofslang, user's level of confusion, user's ability to hear, etc. In someembodiments, the vehicle can conduct a hearing test for automaticcalibration of interaction voice. The hearing test can include soundoutputs from various speakers, various tones, graphic displays, orcombinations thereof. In other embodiments, a user's current region,e.g., part of a country or world can be detected and used to makedeterminations and/or learn behavior or match behaviors from thirdparties. In some embodiments, historic GPS data can be used to determineif user lives in a certain area or just driving by. In these variousexamples, vehicles communicate with cloud processing to exchange data.In some embodiments, processing is conducted by the server or serves ofcloud processing. In some embodiments, processing is conducted byelectronics of a vehicle. In some embodiments, processing is conductedby a smart device in or associated with the vehicle. In still otherembodiments, processing may be shared between the server and vehicle,the server and a user's smart device, or between or among multipledevices that are local and/or remote.

FIG. 3 illustrates a system and use diagram, of the configuration of theuser interface, in accordance with one embodiments of the presentinvention. As shown, applications 114 may be selected by a user thatwishes to generate a custom configuration for a vehicle. In oneembodiment, the user may go to a vehicle website 190 where the user mayselect a vehicle system component that matches the vehicle that the userwishes to customize In one embodiment, the user would establish a useraccount in a cloud service of the vehicle manufacturer, or a third-partysite that provides customization features for the vehicle manufacturer.

The cloud services 120 may provide interface customization 206 toolsthat will allow the user to select the application 114, select thesystem component for the vehicle, and arrange the selected applications,arrangement of the applications on the display screen, settings for thedifferent applications, etc., to thus define a custom configuration forthe user interface. The custom configuration will then be saved to auser profile database 112, which saves the custom configuration andprovides access to the custom configuration for updates from time totime by the user, or for updates provided by the vehicle manufacturer.

In one specific example, a user 121 can visit a website, an app, or aportal to customize a vehicle display 210 using tools provided by awebsite that allows the customization. The tools can include pull-downmenus, selection icons, text entries, radio buttons, arrangement andcustomization feature selectors, program settings, etc. The user canaccess the website using any user device. The user device can alsoinclude setting the custom configuration via a vehicle 160. In general,the configuration can be made using any device that has access to theInternet.

In operation 220, the user will select a vehicle using the toolsprovided by the website. Selecting the vehicle will allow the correctselection of the system component for that vehicle, and any otherupdates or parameters defined by the vehicle manufacturer. The systemsfor the vehicle user interface will then be identified in operation 222.A tool than be provided to allow selection of the apps to add to theinterface in operation 224. As mentioned in this disclosure, the usercan select any number of applications to add to the customconfiguration. From time to time, the user can select additionalapplications to add to the custom configuration or removed from thecustom configuration. In operation 226, the user customization for theuser interface will be received including the defined applications andsystems.

In operation 228, the custom configuration will then be generated andcan be assigned to the user profile of a user account, in a database(s)of websites handling the cloud services 220. In some embodiments, thewebsite may be hosted in a distributed manner, using virtualization anddistributed data centers. The distributed data centers can thencommunicate data and process operation to the vehicle to execute theapplications and system components, and provide resources fromthird-party applications and applications over the Internet.

The generated custom configuration can then be transferred to thevehicle 160 and operated using vehicle electronics 161. Vehicleelectronics 161 can also include a display. As mentioned above, thedisplay can be a single display or a plurality of displays. The displaysare configured to generate images for various screens, selections,icons, buttons, controls, and received touch input and communicate textinformation and other data to users.

FIG. 4 illustrates an example of user 121 interfacing with a display 162in the dashboard of vehicle 160. In this example, the display 162 willproduce a user interface 250 that requests the user to input a user ID.The user ID can be any credentialing type input. The credentials caninclude usernames and passwords, keys, alphanumeric entries, biometricinputs, voice inputs, retina scan inputs, fingerprints, facerecognition, etc.

In one embodiment, fingerprint readers may be integrated intoelectronics of the vehicle. The electronics may perform analysis offingerprints for matching purposes generally by comparison of severalfeatures of the print pattern. These include patterns, which areaggregate characteristics of ridges, and minutia points, which areunique features found within the patterns. It is also necessary to knowthe structure and properties of human skin in order to successfullyemploy some of the imaging technologies. A fingerprint sensor, which maybe integrated into a vehicle surface, vehicle interface, or the like, isan electronic device used to capture a digital image of the fingerprintpattern. The captured image is called a live scan. This live scan isdigitally processed to create a biometric template (a collection ofextracted features) which is stored and used for matching. Sometechnologies that are usable include optical, capacitive, RF, thermal,piezoresistive, ultrasonic, piezoelectric, and MEMS.

In one embodiment, retina scan inputs use scanners that are integratedinto a surface of the vehicle. In one embodiment, the scanner may beintegrated into a rear-view mirror of a vehicle, a dash-board of avehicle, a steering wheel of a vehicle, or some surface that can havesome directional light of sight toward a face of a driver. The humanretina is a thin tissue composed of neural cells that are located in theposterior portion of the eye. Because of the complex structure of thecapillaries that supply the retina with blood, each person's retina isunique. The network of blood vessels in the retina is not entirelygenetically determined and thus even identical twins do not share asimilar pattern. A retinal scan is performed by casting an unperceivedbeam of low-energy infrared light into a person's eye as they lookthrough a scanner's eyepiece. This beam of light traces a standardizedpath on the retina. Because retinal blood vessels absorb light morereadily than the surrounding tissue, the amount of reflection variesduring the scan. The pattern of variations is digitized and stored in adatabase of the vehicle and/or a database of the cloud processingsystem. If stored on the cloud processing system, this identifier may beshared or used with a user account to access multiple vehicles, e.g., inaccordance with permissions set in a profile of a user having theidentified biometric.

In FIG. 4 , user 121 will enter the user ID which would then send acustom interface request to cloud services 120, over the Internet. Asmentioned above, vehicle 160 is connected to the Internet, or isconnected to the Internet at particular times. When the vehicle 160 isconnected to the Internet, the request can be sent to cloud services120, to request the custom configuration for the user.

A user having an account with cloud services 120 will have previouslydefined custom configurations that may be downloaded or accessed withoutdownload for the specific vehicle. The vehicle ID would be sent to thecloud services 120 by the vehicle upon sending the request for thecustom configuration.

FIG. 5 illustrates how the custom configuration 200 that provides theinterface defined by the user is downloaded to the vehicle electronics161 and the display 162 of the vehicle 160. The display 162, as notedabove, is only an example display, and display can be of any size andcan include multiple displays. For simplicity, a single display is shownin FIG. 5 .

In this example, the display 162 is populated with user interfaces forthe system as well as the applications. As shown, app interfaces may bepresented in specific locations in the user interface as well as systeminterfaces that are provided in other specific locations in the userinterface. In one embodiment, the definition of where the specific userinterfaces for the systems and the apps are to be defined is set by theuser during the configuration process.

In other embodiments, the positioning and layout or arrangement of thespecific components of the user interface, whether they are systeminterfaces or app interfaces, may be custom arranged by the system overtime based on use patterns. The use patterns of the user can be learnedby the system so as to arrange the various system components and appcomponents in various locations of the display 162. In otherembodiments, certain interfaces will be surfaced (e.g., shown orpresented) on the display at certain times of day, certain times of theweek, certain times of the month, certain times of the year, etc. Betterplacement of the app components and system components, and programmingof data into the components can be optimized over time based on learningthe input patterns provided by the user to user interface.

For example, if the user always views the weather in the mornings at 8o'clock or 9 o'clock and a.m., then the weather icon or interface willautomatically start to be surfaced (e.g., show or displayed, orillustrated (visually or audibly) on the display during those times. Ifthe user plays rock 'n roll rock music on the weekends and classicalmusic during the weekdays, this preference will also be learned.Learning of these preferences will act to custom define the layouts andplacement of the icons and user interfaces on the display over time. Instill other embodiments, the specific placement, location, andarrangement of the apps, system components, buttons, controls, etc.,will be preset and fixed by the user based on predefined settings.

These predefined or learned settings can be saved to the database incloud services and associated with the user account. Updates to thesettings can then be made at any time by accessing cloud services overthe Internet using any device, whether the devices are in the car, ofthe car, a portable device, a home computer, a work computer, a tablet,a smart phone, a smart watch computer, etc. Also shown in FIG. 5 is anembodiment where a user's smart phone or mobile device is synchronizedwith the user interface of the vehicle 160. In this embodiment, theuser's portable device 210 can synchronize and upload content and UIcontrols from applications stored and running on the portable device210. This provides for safer driving, as the controls shown on thevehicle display can be restricted based on driving or operation status.

In one embodiment, the user can custom configure to have content fromapplications running on the portable device 210 to be displayed in thevehicle displayed 162 in a specific location. This location on thedisplay can then be synchronized or mirrored to that part of the displaybased on the configuration. In still other embodiments, the customconfiguration can determine to synchronize an application running on theportal device to occupy the entire display 162. For example, if the userwishes to use his own telephone calling interface and contacts that arestored on the portable device 210, that information can be populated andmirrored to the display device 162, while still using other systemcomponents or other applications of the vehicle in the background or ina separate screen that is not currently active. In this example, theportable device 210 as well as the vehicle electronics 161 cancommunicate with cloud services 120 at the same time, or when specificfunctions, data or communication is required.

As noted above, systems can be configured to enable local communicationwith mobile devices that may be in the vehicle. The embodiment may beprovided by allowing synchronization with the computing system of thevehicle, or with the computing communications of the portable device.For example, the local communication can be paired automatically, basedon a preset pairing process where pairing keys are entered or present.This provides for automatic settings and synchronization when the userenters the vehicle with the portal device. In some embodiments, userinterfaces associated with applications loaded on the user's portaldevice can also synchronize to the display screens of the vehicle, aspredefined by the user.

FIG. 6 illustrates an example of synchronizing other applications to avehicle display, which may be customized by the user. As shown, 602shows a number of data collection and data interface modules, that caninterface with third-party applications or applications executed on theInternet by cloud services or third-party cloud services. As shown, inthe case of an electric vehicle, data associated with charge unitlocation data 620 can be collected from charge unit install points 608.The charger unit install points can also be providing charging discounts606, which can then be transferred to data manager 602. Traffic data 222can also be collected, whether the vehicle is electric or nonelectric.

Charge grid load 624 data can also be collected, for example forelectric vehicle data. Charge grid load 624 can obtain data from a gridpower demand source 610, which can include power company's localutilities and the like. Route based discounts 626 can also be providedto the user, by collecting mapping data 614 as well as discountsprovided by goods and services providers in the marketplace. Mappingdata 630 can also be managed, to monitor the location of the vehicle inrelation to goods and services that may be provided when the vehicle isin proximity In some embodiments, discount data, advertisements, sales,goods and services offers, etc., can be provided to the userautomatically based on the vehicle's location.

In other embodiments, the user can provide settings in the userinterface that identifies which type of offers or discounts orinformation the user wishes to receive. In some embodiments, alertsregarding offers and discounts can be provided to the user in an audiomanner, to avoid driving distractions. Live traffic data 616 can also beprovided to the data manager 602, as well as geo-data 612. The datamanager 602 is in communication with cloud services 120, to providethese services to computer 640, smart devices 642, remote location 644,and a display of the vehicle 162.

The display the vehicle can be interfaced with logic that runs onvehicle electronics 161. The vehicle of products can include memory andprocessors that execute instructions, operating systems, API processing,application management, telecommunications, network accessing, localcommunication with wireless devices, and general communication with theInternet. Route request can also be provided at the demand of the uservia the display 162, and instant routing 604 can provide routes to theuser based on data collected and managed by data manager 602.

FIG. 7 illustrates example where the user 121 holding a portable devicecan synchronize data from the portable device directly with the display162 of the vehicle. The display the vehicle can be a display on the dashof the vehicle, or any other location as mentioned in this disclosure.As mentioned herein, the vehicle electronics 161 will be provided withcommunication with cloud services 120 provide access to the customizedisplays, customize settings, and customized services provided to theuser as a vehicle drives.

FIG. 8 describes a system in which a user interacts with a model viewcontroller software environment 800 useful for processing APPS usingAPIs 130 on vehicles with vehicle operating systems 129 capable ofprocessing computer code. The model view controller paradigm 800 showsbasic interaction, control, processing, and updating of data useful formanipulating and viewing resulting actions by ta vehicle running an APPin such a system. Such a system useful for running APPS on vehicleoperating systems will accept inputs by a user 121, cloud services 120via data streams, vehicle systems feedback and data streams 812 used bya controller 804 that may constantly poll electrical, capacitive andphysical sensors, and input streams to detect if interactions 808 suchas network passive updates, network active updates, user touch, userspeech, user input, user selection among others has been triggered.

Each input 804 will then trigger manipulation of the system's model 802portion of the APP software paradigm thus invoking stored routineswithin APPS 104 which then in turn interact with the vehicle's APIsystem 130 built upon the vehicle's operating system 129. Depending onthe app presented to the user 121, the input may trigger stored routinesor functions on APP software or operating system level restricted storedroutines or functions.

After the processing of stored procedure code is manipulated witharguments provided by the controller 1804 inputs, visual and or sensoryresults are presented to the user in the view 1806 portion of the modelview controller paradigm. These sensory outputs, data streams,electrical signals may all be translated as additional options, results,dynamic updating, and audio or visual graphical user interface changes810 on any of the user's connected display devices. The user will noticethese results visually or audibly but may also feel or detect changes inthe vehicle's mechanical systems. Updates from the model 802 may also beused to toggle vehicle settings 814 which in turn may invoke changes inthe vehicle's physical, mechanical and electrical systems 128.

Then, the system controller 804 may receive additional updates from thevehicle systems affected or additional user 121; cloud services 120,vehicle systems feedback inputs 812 to re-engage the user in a cyclicalfashion. If no inputs are sensed, the system's controller will continueto poll it's electrical and data I/O systems for input on a continuousbasis.

The model view controller paradigm 800 described is one example of thesoftware input output lifecycle that may be used to invoke, manipulate,process, update portions of computer readable code such as APPS 104using an intermediary API 130 to communicate with the vehicle'soperating system 130. However, APPS 104 may be run on physically wired,wirelessly connected or remote devices having processing abilities totranslate the computer readable code in APPS into actionable invocationson one or more vehicles in order to facilitate or utilize the vehicle'selectrical and mechanical systems in prescribed or customizablefashions.

FIG. 9A describes how vehicle on board computer with input out/putsystem 1900 useful for accepting input, processing input and displayingresults in conjunction with stored computer readable programs orfunctions in the forms of APPs 104 may be structured. Although system900 describes one way to provide vehicle on board computing power to runAPPs 104, the arrangement of the vehicle computer 906 may be altered orarranged in differing fashions with differing connection routing inorder to achieve the same. In this example, vehicle on board computer906 may be comprised of components such as the network interface 910,memory 912, a central processing unit 1914, an input output bufferuseful for streaming data 916, storage 908 having the ability to storecomputer data in long term or short term fashion useful for storedcomputer code procedures in the form of an operating system 129,intermediary stored procedure code in the form of APIs 130, storedsubsets of computer code procedures APPs 104 interacting with API 130 asan intermediary to the operating system 129.

In this example, the vehicle computer 906 has the ability to transmit,receive and process information using wired or wireless connections. Onesuch wireless connection is provided by a wireless data sending andreceiving antenna 928 connected to a network interface 910 useful forpairing with and communicating data with portable or stationary wirelessdevices which may or may not be part of a network 902. Such wirelessdevices include but are not limited to wireless displays 210 b, portablesmart phones 210 a, portable computers, 210 c and even stationaryobjects, structures, buildings, toll bridges, other vehicles etc. Thevehicle's network interface 910 through antenna 928 may also communicatewith cloud services 120 to receive instructions from a remote locationthat invokes stored programs such as APPs 104 on the vehicle's computer.

The vehicle may also send and receive data wirelessly in order toestablish a connection with a peer-to-peer ad-hoc network. Invocationsmay result in output data streams interpreted by wireless devices 210 b,210 a, 210 c as well as wired devices such as wired displays 210 d orvehicle integrated display devices such as windshield heads up projecteddisplay or integrated glass displays 210 e. All data streams generatedby APPs 104 stored on the vehicle's computer may also be triggered bywired devices such as vehicle sensors 918, vehicle electrical systems920, vehicle electrical systems 922, engine control systems 924, vehiclediagnostics systems 926, user input as well as environmental input.

A user and or vehicle may find system 900 useful in one example, wherethe user drives the vehicle past an electronic toll bridge where a feeis required to pass the toll bridge. The vehicle's computer willcommunicate wirelessly as it passes the stationary structuretransmitting and receiving information with it as it drives by. Theuser's vehicle may have an APP 104 installed on the vehicle computer 906that can process the input using the computer's wireless antenna 928,network interface 910, input output system, 916 automatically respondingto the toll bridge with payment information. Once the payment isreceived and processed, the APP 104 receives information from thestationary wireless toll taking device which is then stored eitherlocally on the vehicle's storage 908 or remotely using cloud services120. The results of the transaction are then sent via data stream fromthe compute code running on the APP 104 to a display device(s) where theuser can visually confirm that the toll was paid, accepted and show theuser's remaining balance all through the GUI displayed for APP 104.

FIG. 9B describes one example of how stored data and functiondeclarations may be compiled to provide intermediary access to avehicle's computer controlling vehicle systems 950. Such routines, dataand functions may be arranged in such a way that limited access is givento third party code on APPs 104 to manipulate certain unrestrictedoperating system functions and vehicle systems. Such a method ofproviding the intermediary allowed stored function set to third partycode can be referred to as an API 130.

In this example of an API 130, computer readable code is arranged insuch a fashion that the type of API is described and in this case, anAPI that allows third party control of the vehicle's HAVC system isdeclared. A declaration may be useful for reserving the vehicle'scomputer long term and short-term memory in order to run storedprocedures. The shown declaration 954 describes an example set of datathat may reference memory locations and their contents. The contents ofthis memory location may be modified by stored procedures 956 orfunctions.

This HVAC API 130 has the ability to store data relating to thevehicle's temperature, status, target temperature, split zone temperate,data from electrical and mechanical sensors, calendar dates, and errorinformation among others. Invocable functions 956 are the methods bywhich a third party APP 104 may manipulate data 954 on board a computer.Free access is not given to the restricted data on a vehicle's computer,thus a structured method or methods are described for user by thirdparty APP developers. These functions 956 that may or may not takearguments in order to execute may include functions in the example HVACAPI that update temperatures for both the left and right or given zonesof the vehicle, toggle are conditioning, allow visual skins on the APPGUI, manipulate schedules and displays etc. The HVAC API 130 describedis one example of how one API can control one vehicle system. There maybe variations of the APIs for multiple vehicle systems or one supersetAPI that allows access to all of the vehicle's systems through storedprocedures or methods.

FIG. 9C describes a set of computer readable and executable code 970that can be compiled together by a third party APP 104 developer in theform of an APP. The APP 104 uses structured programming languages toexecute stored functions allowed by the vehicle's system API 130. Inthis example, the APP is a third party HVAC app that allows a GUI to bedisplayed to a user giving them the option to adjust the temperature onthe left or right side of the vehicle up or down. In this case, theAPP's GUI has provided a data stream to the APP letting it know that theuser has selected to set the temperature on the left side of the vehicleto 80 degrees and the right side of the vehicle to 76 degrees. The APP104 will then use functions available from the vehicle's API 130 tomanipulate the data on the vehicle's storage system which in turn willbe electrically polled by sensors, data streams etc. to manipulate thevehicle's electrical and mechanical HVAC systems. The user will noticethe result visually by the data provided by the APP to the GUI as wellas environmentally as the temperature is changed in the vehicle.

FIG. 10 describes the stepped flow of events 1000 as a user interactswith an APP 104, in this case, an HVAC APP 104. The GUI shown for APP104 describes the first step 1002 where a user physically interacts witha sensor, screen, voice system etc. polling to see if an input has beenreceived. The user's input in 1002 has been interpreted by the app toraise the temperature on the left hand side of the vehicle to 80 degreesand maintain the temperature on the right hand side of the vehicle at 76degrees. This input invokes step 1004, which calls a stored function onthe APP 104 that is API 130 allowable with arguments.

The stored function may invoke other helper or associate functionswithin the API 130 in step 1006, which all in tern invokes restrictedcomputer readable code at the operating system and or kernel level instep 1008. These invocations will then in turn command mechanical and orelectrical systems in step 1005 in order to achieve the requestedresponse in step 1002. The results of the commands on the vehiclessystems are based back to the vehicle's operating system or kernel levelin step 1012 which then updates data on the API 130 in step 1014 thatthe APP 104 is polling, such as updating the display to show theresulting temperature in step 1016. The results of a function that isinvoked at the API 130 level updating the display produces a data streamtranslatable and displayable by the vehicle's screen showing the APP104's GUI in 1018.

FIG. 11 describes further example ways an APP 104 may take, process andproduce results 1100. FIG. 10 shows a way to interact with an APP 104locally but a vehicle computer system may relay data, inputs andinformation to the APP while connected to a wide area network, localarea network, cloud process 120 or private cloud. A remote action toinvoke change on an APP 808 may be initiated via a network and pass tothe APP running on the vehicle 160 using the vehicle's antenna 928 orwired interface. An APP 104 running virtually on a network or cloudservices 120 may also take input remotely and process the resultsaccordingly.

Some of the inputs and results 1102 that an APP can take and producelocally or remotely include but are not limited to the set 1104 that canreceive an action, react to an action, control an action, manipulatedata models, report changes to a view or GUI, record events orincidents, learn the types of requests being submitted, learn the timesof request being submitted over time, learn the days of the year therequests are being submitted over time, generalize and interpretrequests, assume user intent in order to automatically invoke changes,automatically and pre-emptively act on behalf of a user, fine tunelearned user behavior etc.

The learned behavior (e.g., learned settings that provide for automaticprogramming) can be assigned to particular applications, particularsub-features of applications, to particular native system features ofthe vehicle, or combination of one or more thereof. The learned settingscan also be managed via an interface, which shows to the user settingsthat have been learned and provides the user with options to modifylearned settings. The modifications of the learned settings can be madevia the vehicle display or any other device having access to cloudservices. The learned settings can also be communicated to the user vianotifications. Such as, “We noticed you like your truck temperature at 3pm to be 60 degrees? Please confirm,” or “We noticed you like your cartemperature at 8 am to be 75 degrees, this will be preset for youautomatically,” or “We have detected your favorite settings, pleaselogin to your account to see settings we have programmed for you or makeupdates,” or other similar notifications via the vehicle or to anyconnected device over the Internet.

In other cases, notifications may not be sent. In some cases, thesettings will just occur automatically. In some cases, the settings canbe manually adjusted by the user way from the auto settings. In suchcases, the manual setting can be learned and can be provided moreweighting since the user took the time to correct an auto setting. Thus,various levels of weighting or importance can be given to learnsettings.

FIG. 12 describes an ecosystem where an APP 104 in conjunction with avehicle API 130 may work together to make assumptions, make decisionsand take actions 1200. API and APP code together can be arranged in sucha fashion that creates an assumption and reasoning logic module 1216.This Assumption and reasoning logic module can take inputs from varioussystems and data streams including but not limited to GPS 1206,calendars 1208, traffic conditions 1204, local news 1202, past data ofuser behavior and interaction 1210, vehicle diagnostics 926, userpreferences 1214, user login profiles 506, environmental interpretationsby sensors 1212, marketing deals 1224 among others. These inputs can belocal and physical or remote and abstract via a network. The assumptionand reasoning logic module 1216 compiles data from these sources toinvoke decisions and actions on a decision and action engine 1218.

This decision and action engine 1218 has the ability to execute on whatthe assumption and reasoning logic module has determined needs to bedone. The decision and action engine has the ability to produce alerts,both local, on screen, audibly, visually or remotely on a remote displaydevice 210 a-e using a data network. The decision and action engine 1218also has the ability to change vehicle controls automatically on behalfof a user without user action based on assumptions made by theassumption and reasoning logic module 1216. Additionally, the decisionand action engine has the ability to request a decision from the userpreemptively in order to change vehicle controls.

This may be achieved locally or remotely requiring input from a user toproceed. For instance, the assumption and reasoning logic engine hasdetermined that the user may want to have his or her car automaticallystarted at 7:55 am because the user typically starts the car at 8 am.Starting the car at five minutes early will allow the system to heat thevehicle to the user's typical liking. However, the assumption andreasoning logic may have only reached a level of confidence of 75% where80% confidence is required to act without user input. Thus, the system,being only 75% sure that the car should be turned on will automaticallysend the user an alert requesting a decision on whether or not to turnthe vehicle on. Once the user 121 provides a decision remotely on theirremote device 210 a, the decision engine 1218 updates the assumptionmodule 1216 so that it can augment its assumptions for an updated levelof confidence on the next action trigger. These actions by the userautomatically and continually update the assumption and reasoning logicmodule 1216 in order to fine tune the level of confidence on actingwithout user input and learn the user's behavior for future decisions.

FIG. 13 illustrates an example of user interfaces and interaction modesprovided on or for various displays of a vehicle, in accordance with oneembodiment of the present invention. In this example, the vehicle isshown communicating with cloud services 120, utilizing the vehiclecomputer. As described above, the vehicle computer can includecommunication circuitry to enable wireless communication with theInternet and servers that can provide cloud services 120. In thisillustration, the user interfaces or displays of the vehicle are shownwith graphically rendered gauges, information, and data that may berelevant to the user of the vehicle.

In one embodiment, the user the vehicle may log into the vehicle or bepaired to the vehicle automatically so that a user account of the userprovides the preferences of the user for displaying select informationand communicating with cloud services 120. Cloud services 120 cancommunicate with other Internet data sources, and cloud applications ofthe user, such as calendars, appointment books, reservations, websites,merchants, mapping applications, discount providing applications, chargelocation services, payment services, parking services, vehicle avoidanceservices, etc.

Continuing with the example of FIG. 13 , the user interface provided inthe main dashboard in front of the steering wheel has been rendered forthe user account of the user in accordance with the user's interactionmode selection. As will be described below, the interaction modeselection will allow the user to either custom configure or select fromcustom configurations the type of information that would be rendered onthe displays of the vehicle or provided via audio output of the vehicle.In this example, the interaction mode for the user account is one thathas been selected to reduce the amount of clutter provided in the maindashboard interface.

As used herein, dashboard clutter or display clutter refers to when toomany gauges, icons, information, GUIs, meters, text, pop-ups, colors,designs, animations, etc., are rendered on the displays, and which maycause distraction while the user is driving. Reducing the amount ofclutter is a feature for vehicles that provide interactive displays thatcan populate so much information that a driver may become distracted. Inone implementation, the level of information that may cause distractionwill vary, as this is a subjective metric that is personal to eachuser/driver. In some embodiments, the amount of information provided tothe displays can be dynamically changed based on the condition of thedriving or non-driving of the vehicle. For instance, if the vehicle isnot being driven, more information can be rendered on the displays forthe user.

If the vehicle is parked or at a stop sign or stop light, moreinformation may be rendered on the displays. When the vehicle isoperationally moving, less information would be rendered on the displaysso that clutter can be reduced. In one embodiment, more or lessinformation or icons or gauges may be displayed or shown on the displaysin a fade in and fade out fashion, so that the instant appearance ofgauges will not be distracting sight for the driver. In one embodiment,when reduce clutter displays are provided, basic gauges for operatingthe vehicle or required by law will be required to stay viewable in thedisplays while the vehicle is being driven or is in operation. Forexample, a speedometer gauge is required or is vital to the driving ofthe vehicle, and therefore such gauges would not be removed or not shownwhen reduce clutter displays are selected. It should be understood thatreduced clutter is subjective, and the interaction modes provide fordifferent types of modes and modes that can be customized or customizedover time, such that the level of information is not distracting to theparticular user, from a personal and subjective view point.

In one embodiment, the information that is provided on the displays canbe dynamically set based on the context of the vehicle's state, theuser's calendars, the weather, and other factors. In one example, thefuel gauge shown in the main dashboard display of the vehicle in FIG. 13is shown to include a fuel gauge. The fuel gauge in this example isshown to have appeared on the dashboard display because the vehicle'sstate is that the fuel is low and requires refueling. In one embodiment,the vehicle computer can be communicating with cloud services 120, whichwill automatically identify information regarding available fuelingstations nearby.

For example, one of the displays of the vehicle shown in FIG. 13illustrates that contextual information can be provided as arecommendation, which identifies that a gas station is within 0.25 milesof the current location of the vehicle. In addition, a mapping serviceor map program of the vehicle can be automatically displayed on one ofthe displays of the vehicle showing the location of the gas station(e.g., Ted's Gas). Accordingly, the information being displayed on thevehicle is contextually related to the state of the vehicle, thelocation of the vehicle, and applications are automatically loaded andprovided for generating information relevant to the vehicle and itsstate.

FIG. 14A of displays of a vehicle, which are cost to interactively showdisplay items based on the context of the vehicle and the context of theuser and preferences of the user. In one embodiment, the preferences ofthe user can be learned over time by examining use patterns of the user,which are signals indicating actual preferences by the user. In otherembodiments, the patterns of use, interaction, preferences, inputs,memberships in loyalty programs, shopping history, prior use ofdiscounts, and other information can be used to identify what type ofcontextually related information should be displayed to the user aced onthe current state of the vehicle and the current state of the user andthe geo-location of the vehicle.

In this example, the type of information that is surfaced to displays ofthe vehicle can depend on the context of information associated with theuser, who is logged into the vehicle by way of a user account that isconnectable to cloud services 120. In this example, certain informationcan be examined to determine what type of contextual recommendation canbe provided to the user, and what type of contextual information can beprovided based on learned behavior of the user, which provides aprediction or likelihood of the type of information that may be mostrelevant to the user in the particular context. In one embodiment, aserver may determine what data to send to the vehicle, and when sent,the data can be presented on a screen or audio output of the vehicle.

In one implementation, the data sent can include a time threshold thatidentifies (e.g., identifier or tag or data) when the data can bepresented or when the data may no longer be presented. The timethreshold can be in the form of data, a tag, a marker, an identifier,flag, or the like, which identifies when the data should no longer bepresented (e.g., data may become of context and thus no longercontextually relevant). For example, the data may be relevant for aparticular time, window of time, or period of time. If the period oftime has passed, the data can be considered stale, such that the data isno longer allowed to be shown, even if the vehicle received the data.For example, the data may be sent to the vehicle when the user isactively using some display or input, and the use of the display orinput may prevent the sent data from being presented. In this example,the data is held (in memory) until the user has completed the input (oruse of the vehicle, or personal device that may be active (e.g., phone,tablet, directions device, etc.)) to avoid interruption.

Once the interaction has been completed, vehicle software and/orelectronics can determine that the data received is no longer valid,stale or no longer relevant to the current geo-location, personalpreferences, vehicle condition, or some other factor. In oneimplementation, therefore, data that is sent to the vehicle forpresentation or surfacing may not be presented, if other localconditions, user conditions, and/or geographical conditions determinethat the data is stale or no longer useful. Thus, gating the data frompresentation, even after receipt by the vehicle, enables forpresentation of possibly un-needed data to the user, thus reducingdistractions.

In another example, contextual information that may be viewed mayinclude them on a fuel that remains in the vehicle at the particulartime (which is a state of the vehicle, among many different types ofstates of the vehicle), the day of the week, whether the day of the weekof the holiday, information from the personal calendar, historicaltravel times during the time of day, the time of day, loyalty cards thatthe user may hold or like, traffic information associated to the currentgeo-location of the vehicle, the current weather, learned past behavior(when the user likes to stop for coffee), nearby coffee shops (coffeeshops being a learned type of good liked by the user), discounts locatednearby, discounts located nearby other services that are needed at aparticular point in time, and other factors.

These contextual types of information associated with the user, thevehicle, the number of passengers in the vehicle at the time, the user'scalendar, the users likes, the users past interactions, the predictionsof what the user wishes to see or may want, etc. are only but a fewexamples, and are shown without limitation.

Continuing with the example of FIG. 14A, based on the contextualinformation obtained by the vehicle computer from cloud services 120 andfrom information stored in the vehicle computer or obtained from acomputing device of the user, determinations can be made as to the typeof contextual recommendation that may be surfaced to a display screen ofthe vehicle.

It should be understood that surfacing too much information can causedistraction while driving, so therefore contextually relevantinformation that is predicted to be needed or wanted at a particularpoint in time should be displayed as a contextual recommendation. It isbelieved that the information that is automatically being contextuallyprovided to the user on the displays is information that would have beensearched for by the driver.

Thus, by providing the intelligence surfacing of contextual informationto displays and or audio outputs of the vehicle, less distraction willoccur because the driver will not need to interact with user interfaces,but instead the information will be provided to the driver just as thedriver will need the information. For example, the drivers beingprovided with information to the closest Chevron station which is 0.45miles away from the current location, a map to the Chevron station isshown, and a coffee coupon is also shown on the display.

The coffee coupon is provided to the user because the coffee shop islocated near the Chevron, and the user typically purchases coffee duringthis particular point in time and the coffee shop is next to the Chevronwhere the user will likely wish to purchase gas based on his ownershipof loyalty cards for Chevron. As such, this information has beenprovided to the user at time when the user would want or need theinformation, which cuts down in screen clutter and also reducesdistracted driving.

FIG. 14B illustrates yet another embodiment of contextual surfacing ofinformation to one or more display screens of a vehicle or surfacing ofaudio to the vehicle. In one embodiment, the user prefers to have a lowclutter screen, wherein the display panel in the dash has few items,such as vital gauges needed for driving the vehicle. In one example, ata minimum, a speed gauge is provided. As the driver drives around, fromtime to time, depending on the context or state of the vehicle, certaingauges may be surfaced to a display.

As shown, an RPM (revolutions per minute) gauge may surface gradually onthe main dashboard display when it is determined that the vehicle's RPMsare too high. The gradual surfacing, in one embodiment, allows forreduced distraction of the driver. This is because fast surfacing ofgauges or icons on the display screen may distract the driver to lookdown and away from the road. However, when the gauges are surfaced in afade in from light gray to full color or contrast, the driver willnotice the newly surfaced information gauge when the driver next looksdown at the gauge.

In the example of FIG. 14B, it is also shown that the contextualinformation regarding the user, the vehicle, the geo-location of thevehicle, the time of day, the day of the week, and information found inthe user's online calendars and to-do lists can be mined to providecontextual recommendations. As shown, as the vehicle is communicatingwith cloud services 120, contextual information can be provided to thevehicle display(s). The contextual information, at the particular timeof day and when the user is driving or activating the vehicle, isprovided when the user is likely going to need the information.

For instance, the current to-do list shows that the user needs a hammer(and other things), and because the user is near a store that sellshammers, that information can be provided to the user. The informationis provided or surfaced to the user by presenting it on a display, aftercloud processing determines that other contextual parameters suggestthat the user would be interested in a particular good or service, e.g.,in this example a hammer, at that particular time of day and day andparticular geo-location. In addition, learning systems also candetermine that the user usually likes discounts, so special discountscan be automatically searched for from various online and off-lineretailers, and the discount or coupon can be surfaced to the vehicledisplay at the time the contextual information is provided to the user.

As shown, in addition to presenting the distance to the user, providinga map to the store location, the discount is presented on the display.The discount can be provided for use by the user in various ways. Insome examples, the discount can be automatically sent to user's device(e.g., smartphone, tablet, watch, etc.). The discount can be in the formof a digital coupon, a code, a link, or some other identifiable form. Instill another example, the coupon can be provided to the user when theuser selects it on one of the displays of the vehicle. The coupon canthen be transferred to the user's device, or can be sent to the retailer(with the user's account info), so when the user arrives at the storethe coupon is automatically credited, or can be sent from the serverdirectly to the user's device.

FIG. 15 shows another example of intelligent gathering of contextualinformation for the user and surfacing just the information that isdetermined to be useful to the user, in accordance with one embodiment.In this example, the vehicle is an electric vehicle (EV) that requirescharging from time to time. The system is also configured to connectwith the user's accounts (e.g., for a user that is registered, has auser account and is actively using his or her account when occupying ordriving the vehicle). As in other examples, the main dash of the vehiclehas a screen that is configured to render icons, gauges, displays, data,and other vehicle information. In one interface mode, low clutter isselected.

The low clutter selection (e.g., selectable as an interaction mode) isconfigured to present very few icons or gauges, such as those that maybe required for vehicle operation. In the illustrated drawing, the maingauge that is shown is the speed gauge and an optional digital read outof the speed. As the user drives around or uses the vehicle, it turnsout that the user's calendar determines that an appointment call needsto be made. This determination can be made by reference to the user'sonline calendar or calendar on a device that is shared with the vehicle.The appointment to call is, for example, “Call Ted.”

At the time the appointment time arrived, the user was listening tomusic, but if the vehicle computer/server determines that the callshould be suggested, the vehicle display can change to show “Calling . .. Ted.” Also or alternatively, the main dash of the vehicle can show anicon that is surfaced gradually to the display, which may be anindicator of an audio-interface. The audio interface can, for example,as the user if he wishes to call Ted, and the user can simply answer byvoice input. The voice input can then trigger the activation of the callfor the user. Still further, other parameters, in addition to thecalendar can be analyzed to determine that the context is appropriatefor surfacing the question to the user.

The analysis can include, for instance, processing the informationassociated with the current context of the user, the vehicle, thecurrent time of day/week, historical data, weather, discounts, servicedata, etc. Over time, based on the selections, choices of interfacing,what was selected and when, what was selected when the vehicle wasparticular geo-locations, what was selected and how many people were inthe vehicle with the user, selections or preferences made by passengersof the vehicle, and other data. This data is mined to find overlappingintersections in data and to apply rules and assumptions that formlearned data and patterns. This learned data and patterns are used tobuild a learning database that can grow to include richer data overtime, and can assist in providing intelligent contextual data fordisplay on the displays of the vehicle or for audio output to thevehicle.

It is believed that by providing users with information they need, theywill spend less time making raw user interface selections (e.g.,requiring one or more inputs, taps, touches, swipes, navigations,launching of apps, selection menus, inputs, etc.), which may increasedistraction. In one specific example, data from a user's online datasources can be mined to provide information the user needs andcontextually when needed.

For instance, if the user's email shows that the user has booked airlinetickets and the time of departure is within 2 hours, the user may beprovided with a map to the airport, may be provided with online checkinginterfaces for voice entry, may provide rental car check-in or returninformation, etc. Thus, based on the context of what the user is doing,what the user has planned, when the user has done in the past (e.g.,learning), certain information can be surfaced for consumption by theuser. The result is less distracted driving, and efficient usage of theuser's time.

In one embodiment, information/data that is sent to the vehicle, from aserver, has a higher likelihood of being consumed or used when it isexamined for context. As used herein, the information is likely to beaccessed, consumed, viewed, read, listened to, and otherwise used, whenthe information is sent upon confirming context of one or moredimensions (e.g., geo-location, vehicle state, user history, learnedpreferences, current interaction, etc.). In one embodiment, data that issent, but not yet presented, may lose context, and that data may be comestale. In such cases, data may not be presented by the vehicle, evenafter safe receipt by the vehicle electronics 161. Thus, context can bechecked at the vehicle as well, and/or the data can be received withidentifier data that identifies or tags when such data sent is or may nolonger be valid. Thus, in one embodiment, the vehicle may simply filteror cull out information (e.g., supplemental content) that may no longerbe valid, based on the received identifier or tag, e.g., such as thetime frame of when the data was valid has passed, or the vehicle is nolonger close to a particular merchant, or the like.

FIGS. 16A-16C illustrate several embodiments, where a user has selectedan interaction mode that reduces clutter. The reduced clutter display isshown in FIG. 16A, where nearly all display items are clean or notshown. The display can be provided with wallpaper or skins that reducedistraction or can be turned black or a different color that allows forthe screens to blend-in to the look and feel of the dashboard. Forexample, if the surround parts of the screen appear to be leather orsteel, the image on the screen can simulate an extension of the physicalnature of the dash or vehicle parts. In this case, the cloud services120 may still be monitoring information to determine what to surface tothe display screens.

In one embodiment, traffic data is obtained when the system determinesthat the user would likely be checking traffic information. This may betriggered when, for example, the user appears to be taking longer todrive home after work than normal, or the driver is driving slower thana current speed limit of a road, or a traffic accident is identifiedahead, or based on learned use (e.g., the user typically checks trafficat 5 pm on a workday, etc.).

In FIG. 16B, the displays are shown to be populated with informationobtained by cloud services (or obtained by the vehicle, or obtained by adevice of the user in the vehicle, or combinations of two or morethereof). The system may alert the user that an accident is up ahead.The user, based on account information (e.g., history of user,propensity, or likelihood), may usually select to re-route, so thesystem automatically provides a re-route in the map on the display. Inone embodiment, data for information associated with the geo-location issent to the vehicle when the profile of the user identifies likelihoodfor consumption of the information associated with the geo-location. Anexample may be, without limitation, a user drives by a Chevron gasstation, but the user prefers Teds Gas, so the user will not stop, eventhough the vehicle needs gas and the user is proximate to Chevron. Theuser would be viewed to not have a likelihood to consume informationregarding the nearby Chevron.

If the user's shows that the user does not have appointments or does noturgently need to arrive at the destination, the system may not provide are-route option if the extra distance is more than the user likes todrive. Other contextual information can be mined, including a learnedprofile of the user, which shows what the user likes, does, prefers, hasdone over time as a pattern, etc.

FIG. 16C illustrates example where the user's EV is running low oncharge. The system may surface this information the main dash display(e.g., either gradually, instantly or in a gray-tone or some specialidentifying color. The speedometer may also be slowly shifted (or slide)to the left, as the more relevant information that is surfaced is placednear the center of the main dash display. In one embodiment, the centerdash is considered one of the least distracting locations for driver toview.

Alternatively, the information can be displayed in a heads-up display onthe windshield of the vehicle (e.g., as overlaid or non-overlaid text oricons or graphics, videos, or combinations), which reduce distraction ofthe driver. Continuing with FIG. 16C, other displays also showcontextually relevant information, such as the range remaining for theEV, coupons for coffee near the charge locations, the price of charge,the option to buy ahead of arriving, buying to reserve the EV chargingspot, etc. As noted above, the contextual information is processed byparsing data obtained over the internet, data obtained from the user'shistory profile, data obtained from learned preferences or habits. Insome embodiments, the history profile can itself include learnedpreferences of the user, by way of the user's account.

FIG. 17 illustrates an example of a screen that may be presented in thevehicle or a computing device or on some graphical user interface (GUI),in accordance with one embodiment of the present invention. In thisexample, the type of information presented may be to allow a user toselect an interaction mode to be used for the vehicle. The interactionmode is used to identify the type of interactivity, complexity of datapresented on the displays of the vehicle, types of data that can be ordesire to be sent to the vehicle for consumption by a user, andpreferences that would be liked by particular types of people.

Because people vary in preferences widely, the example of providingdifferent types of interaction modes for particular vehicle willsimplify selection by users so that the interaction mode best fits theirdesired use or intended use of the vehicle. Some people are moretechnology savvy while others wish to avoid technology altogether or ata reduced consumption rate. In the example shown, various modes can beprovided. Although five modes are provided as an example, more or lessmodes can be provided depending on the design or implementation.

The example modes include an intelligent mode, a simple mode, a seniormode, and informed modes, an auto mode, etc. These modes identify thetype of interactivity or the weighted the user wishes to identifyinteraction with his or her vehicle. As noted herein, the modes can beassigned to the vehicle for when the user is present in the vehicle, andneed not be tied to the specific vehicle. For example, if a user accounthas identified the type of mode they wish that operate one or morevehicles, that mode can be transferred to a specific vehicle for aperiod of time. For instance, a user may wish to operate a sharedvehicle and the mode in the users account can be transferred to theshared vehicle.

The user may wish to transfer the interaction mode to a rental car. Theuser may wish to transfer the interaction mode or activate theinteraction mode on a personal vehicle when that particular user is thedriver. In other embodiments, a vehicle can transfer between oneinteraction mode or another interaction mode based on user input. Insome embodiments, the user may choose to utilize a wizard, which willallow the user to his or her type of descriptive behaviors liked ordisliked when operating a vehicle.

These behaviors are descriptive, and need not be specific to an actualsetting. Once the descriptive behaviors are selected by a user, thewizard can identify a specific type of interaction mode which will thenapply a plurality of settings for the interaction mode.

FIG. 18 illustrates one example flow of operations used to providecontextually relevant information to the displays of the vehicle, forone embodiment. In this example, the information is consideredsupplemental content. The content is considered to supplement the user'sdriving experience and reduce distraction, as the supplemental contentis selected based on the circumstances of the vehicle location, theuser's preferences, the user's learned behavior, the user's calendar,the context of driving, the user's travel plans, and other factors asdescribed throughout this document and those documents incorporatedherein by reference.

In one embodiment, supplemental content can include data that isunformatted and is later formatted at presentation, data that is comeswith format data, data that is parsed out of formatted data forpresentation in different forms, data that is obtained from internetservices, such as mapping programs, calendars, news, social,appointments, cloud storage, advertising, and the like. In some cases,the supplemental content can be data that that triggers opening of awebsite or application on the vehicle display. In such cases, most ofthe data will be formatted based on the source formatting provided bythe internet service. In other embodiments, some of the data obtainedfrom the internet service can be re-formatted for display in a nativedisplay format. In some implementations, text data can be extracted frominternet services, such as text search results to for display in acustom or native format for the vehicle displays. In still otherembodiments, the display of the day on vehicle displays can take on ahybrid approach, which may depend on the type of content, application,app, interface, program, license, form factor, content type, and thelike.

In one embodiment, the supplemental information is filtered to provideeven more custom tailored select supplemental content. For instance,even though the user may need gasoline, the user may prefer another typeof gas than that immediately available. Further, even though the userprefers coffee and purchases coffee often at 8 am on a weekday, thatparticular weekday the user may be heading to a client meeting and thedriver appears to be late. Consequently, a coffee coupon or locationinformation may not be provided, as this supplemental content is notuseful to the driver and may only serve as a distraction. In theillustrated example flow of FIG. 18 , operation 1800 includes receivinggeo-locations of a vehicle over time, at a server configured to executecloud services.

The cloud services may be operated by one or more entities, publicentities, private entities, entities such as vehicle makers, entitiessuch as vehicle service provider entities. These entities may operationwith one or more servers. The servers can be individual servers, groupsof servers, services distributed geo-graphically for load balance orimproved quality of service (QoS), servers operated by cloud services,virtualized servers and storage, and combinations thereof. Otherexamples can include processing performed by the servers and someprocessing by the vehicle computer or devices of the user. In stillother examples, apps of the vehicle can communicate with cloud servicesusing user accounts, which provide access to the user's history andprofile.

In operation 1802, the system will access the profile of the user toidentify various data, including identifying a history of use of thevehicle of the user (e.g., the vehicle that is registered to the user'sprofile). In some embodiments, no vehicle is pre-associated to the useraccount or profile, which allows for dynamic transfer of the user'saccount to any vehicle the user steps into and syncs with. Thus, whenthe user access the profile from any vehicle having access to orproviding access to cloud services, the custom services of the user canbe used on the currently driven or used vehicle.

In this manner, the vehicle is not actually associated to a user butinstead, the user's profile or account is associated to the vehicle fora period of time the vehicle is driven by the user. In one embodiment,the profile can also be used on rental cars or other vehicles that areused for short periods of time or on cars that are loaned out by othersfor use.

In operation 1804, from time to time, the system generates a pluralityof learned preferences that are associated to the profile of the user.This association can occur by, for example, by examining the history ofuse of the vehicle (or use by the user on other vehicles). The learnedpreferences can change over time as the user's preferences change oruser input selections made change over time, and based on the particularcontextually related circumstances of when the changes or inputs weremade. Thus, the learned preferences can, in some examples, changeperiodically when enough data is collected to create and define apreference with a threshold level of certainty.

In operation 1806, the system identifies supplemental content fordisplay on the user interface of the vehicle, for a current geographiclocation and for a current time. As noted above, the supplementalcontent can include information that can be displayed on a screen of thevehicle. The information can also be output by voice and the user mayinterface via voice input or combinations of voice input and physicalinput (e.g., touch a screen icon, turn a button, toggle a button, pressa button, slide rolling item, shift a lever, rotate a knob, etc.).

In operation 1808, the system (e.g., server) can filter the supplementalcontent to produce or leave select supplemental content for display onthe user interface of the vehicle for the user. The filtering can bedone, for example, based on the plurality of learned preferences for theuser. As noted above, the filtering is done to cull-out information thatthe user is known not to need or want, or cull out information that isnot appropriate for the time of day or week, or cull out informationthat is conflicting with the user's schedules, or cull out informationthat would conflict with the user's more preferred likes or dislikes.

Further, the culling of supplemental content can also changes over timeas the driver moves around and the geo-location changes. For example, ifthe user's favorite fueling station is nearby, at 8:15 am, and thevehicle needs fuel (but still has enough to driver another 90 miles),but the user needs to take a conference call from 8:15-8:30 am, thesystem will not surface (e.g., cull so this supplemental content is noprovided to a display or audio output) information regarding the nearbyfueling station. Instead, the vehicle will surface and notify the userof the conference call and/or show the option for another fuelingstation that is along the path or near the destination.

In another embodiment, the identification of supplemental content inoperation 806 and the filtering operation of 808 may be processedtogether in one processing operation. That is, the selection of thesupplemental content can itself include the culling for the user and theuser's profile, so as to avoid additional or secondary processing. Instill other embodiments, some users may wish to explore outside of theirprofile, and the user may select to receive 10% of the supplementalcontent outside of his or her profile. This provides a controllableamount of supplemental content that can be surfaced to the user overtime. If the user likes the supplemental content, the user's profile canbe adjusted by the learning of preferences, selections and/or inputsover time.

In operation 1809, the server sends the select supplemental content tothe vehicle over a wireless network. The select supplemental content isconfigured for display on the user interface of the vehicle. The userinterface, as described above, can be any screen of a vehicle, includingscreens interfaces on buttons, toggles, dials, sliders, etc. Thedifferent screens on buttons can show different or customizedinformation depending on the mode selected.

Thus, the same knob, for instance, can provide different functions whenit is turned or touched, depending on the mode. As noted above, theuser's profile can also be augmented or controlled for display ofinformation in various formats, such as intelligent mode, simple mode(easy), senior mode, informed mode, and custom modes selected by theuser, learned over time, or recommended by the computer (let thecomputer decide option).

FIG. 19 shows a flow of one example embodiment, wherein an interactionmode is selected. In this example, operation 1910 includes receiving arequest to access a user account of a user for a vehicle. The useraccount provides access to cloud services. The cloud services areaccessible via one or both of a computing device or a vehicle computer.

In operation 1912, data for display of the plurality of preferencesassociated with interacting with and receiving information from userinterface systems of the vehicle is provided. In this embodiment,preferences associated with the types of display items and types ofinformation the user wishes to see or rendered on the displays/screenscan be provided. This information can be provided in the form of userinterface inputs, table selections, typed in selections, radio buttonselections, selection lists, grid data input, voice input, or any typeof input method that can be captured by a computer, or combinations oftwo or more thereof.

At this point, selections of at least two of the plurality ofpreferences can be received at the user interface in operation 1914. Atleast two selections are useful to receive, so that properdeterminations of the type of interaction modes that are available tothe user can be selected. Alternatively, if the user already knows theinteraction mode he or she desires, the selection can simply be oneselection of an interaction mode, instead of selecting two or morepreferences, which are used to identify an interaction mode.

In operation 1916, and interaction mode is determined for the vehicle(user account) based on the received selections. The interaction modemay define a number of information items to be displayed on a screen ofthe vehicle. Additionally, the interaction mode can identify theintensity of interactive notifications to be received at the vehicle fordisplay on the screen of the vehicle. For example, more intensity caneat equate to more information being displayed or notifications providedto the vehicle. Less intensity can be fewer notifications, orinformation provided to the one or more displays of the vehicle.

The interaction mode, in one embodiment, will define the number ofinteractions that would be provided to the user. If over time the userwishes additional notifications sent to the vehicle, the user maycustomize the interaction mode. In another embodiment, the user cansimply choose a different interaction mode which may globally change theinteraction mode and types of information items displayed and number ofinformation items displayed, as well as the intensity of interactivenotifications.

In operation 1918, the configuration setting is sent to the vehicle. Theconfiguration setting used to implement the interaction mode on thevehicle is data that is sent when the user account of the user is activeon the vehicle. For example, if the user accessing a particular vehiclewishes to implement his or her interaction mode on that vehicle, theuser account of that user will identify and interaction mode, which canthen be transferred to that specific vehicle. As noted above, theinteraction mode may be customized to the user account so that theinteraction mode can be used in any number of vehicles the user wishesto operate. In other embodiments, the interaction mode is programmed toa specific vehicle, such that that vehicle holds the interaction modeprogrammed for continuous use.

In still other embodiments, the interaction mode can be used in aspecific vehicle for a period of time. The period of time can be for oruses of the vehicle, for a week, for a month, for a day, or any numberof identifiable fractions of time. Still further, the interaction modecan be automatically applied to a vehicle when a particular user isidentified/detected to be in or proximate to the vehicle.

Identification of the user in a vehicle can be by way of having the userinput his or her password or user ID into the vehicle electronics 161 oruser interfaces or screens. In still other embodiments, identificationof the user in a vehicle can be by way of biometric identification. Thebiometric identification can be by way of voice input, voice ID, retinascan, finger print ID, gesture ID, or a combination of multiplebiometric identification methods.

FIG. 20 illustrates another flow diagram, in accordance with oneembodiment of the present invention. In this example, method operation2320 includes providing data for display of a plurality of preferencesassociated with interacting with and receiving information from userinterface systems of a vehicle. The preferences being descriptive of ohway of interacting with the vehicle, instead of express settings.

The descriptive ways of interacting with the vehicle can be by referenceto that type of information the user wishes to receive, the mood theuser is in, the familiarity that a user has with technology, the degreeof simplicity desired by a user, easy user interfaces which may beuseful to senior citizens operating a vehicle, etc. Thus, instead ofproviding a plurality of input settings, a number of questions,statements, interactive preferences, can be displayed or asked of theuser verbally so that the user can answer a sequence of questions thatcan be used to then identify an interactive mode for the vehicle.

Thus, this provides an easy way of interacting with the vehicle usingnatural language that does not require the user to expressly entersettings, navigate user interfaces, and the like, which may be tootechnologically complex or uninteresting to certain users. In operation2022, and interaction mode for the vehicle is determined based on thereceived selections. The interaction mode can be one of a preset numberof modes. Each mode may define a plurality of settings to be madewithout requiring the express settings of the individual settings. Inoperation 2024, the configuration settings are sent to the vehicle.

As mentioned above, sending configurations to the vehicle can be bywireless communication. The sent configurations can be set from a serverthat has access to the Internet for transmitting the data to thespecific communication electronics of the vehicle, which in turn allowfor implementing the settings on the vehicle automatically or after theuser has approved the input of the settings. In one embodiment, theconfiguration settings are identified to be a best match of theinteraction mode for the vehicle, for the user, and based on thepreferences that are descriptive of the way the user wishes to interactwith the vehicle.

FIG. 21 illustrates one example of a flow diagram where in interactionmode can be selected and customized for a particular user, in accordancewith one embodiment of the present invention. In one example, theparticular user is a user having an account and an associated profile.In operation 2100, for a user account, a plurality of preferencesassociated with interacting with and receive information from userinterface systems of the vehicle are provided.

The preferences are descriptive of a way of interacting with thevehicle, instead of express individual settings. In operation 2102, themethod includes determining and interaction mode for the vehicle basedon the received selections. The interaction mode is one of a presetnumber of modes, and each mode defines a plurality of settings to bemade without requiring express settings for each of the plurality ofsettings. In operation 2104, the configuration settings are sent to thevehicle. The configuration settings are identified to be a best matchfor the interactive mode of the vehicle, for the user account. Inoperation 2106 and 2108, over time the user will provide input settingsor selections or interactions utilizing the initial or interaction modeidentified or determined in operation 2102.

In operation 2106, information associated with interactions made at thevehicle over time is received. The information identifies changes madeto one or more the settings of the vehicle. In operation 2108,information associated with interactions made at the vehicle over timeis received. The information identifies use of interactive itemspresented on the display of the vehicle. In operation 2110, the receivedinput from operations 2106 and 2108, over time, are provided forlearning patterns of use of the interface items or changes for the useraccount. In operation 2112, recommendations to change the interactionmode or an aspect of the interaction mode of the vehicle is sent to thevehicle.

Information sent to the vehicle can be provided by way of one of theuser interfaces, or audio output, text messages to a user's device,e-mail messages, messages to the user's online account, or the like. Inone embodiment, operation 2114 may include receiving user inputrequesting changes to the interaction mode. The changes requested can bebased on the recommendations provided to the user. For example, therecommendation can be provided to the user suggesting that the userchange the interaction mode from a simple mode to a complex mode or anintelligent mode, or some other customized mode. In another embodiment,the change to the interaction mode can simply be an adjustment to theselected or determine interaction mode.

For instance, the initial interaction mode can remain selected, yet oneor more aspects of the interaction mode can be adjusted or changed todefine a customized interaction mode for the user account in operation2116. In one embodiment, once the customization has been defined, thecustomization settings are sent to the vehicle for implementation. Inone embodiment, the changes in customization can occur on the vehicleitself and over time the customizations can be communicated to a server,which then implement the changes for the user account. In this way, theserver and the vehicle can maintain a synchronized the interaction modeand changes made over time to either the vehicle or the modes in theuser account.

FIG. 22 illustrates one example of cloud services 120, which may becollecting or interfacing with a plurality of data providers, datasources, data processors, third-party applications, third-partyservices, other applications, Internet data, or combinations thereof. Inone embodiment, the vehicle is a connected vehicle which has access tocloud services 120. Over time, the vehicle will traverse differentgeo-locations, which will then be communicated to cloud services 120periodically, continuously, or on some schedule.

In addition, user interactions and input settings 855 can also becommunicated to cloud services 120. This information is communicatedregarding user interactions, such as inputs or settings is also tied tothe geo-location of the vehicle, the time at which the settings weremade, the circumstances of when the changes were made, the contextualrelationship of circumstances to when settings are made or inputs aremade, and the like. As shown, cloud services can include applicationsand logic and other components which are described above.

Additionally, cloud service can include user accounts and databases. Insome implementations, cloud services can be operated by specific serviceprovider, or multiple service providers, a vehicle cloud service, aninternet company, vehicle manufacturers, vehicle service providers,third party service providers, or combinations thereof. Examplecommunications by cloud services 120 are shown, without limitation. Theexample communications can be to geo-location 2360, time of day data2262, local events and news 2264, day of the week calendar data 2266,learn the preferences 2268, explicit preferences 2270, online calendars2272, device calendars 2274, social media data 2276, etc.

In one implementation, at least one aspect of one of the preferences isdata obtained from an internet service. The internet service can be aprivate cloud, a public cloud, website data available via open APIs, orcombinations thereof. The internet service may also be one of a website,or a calendar, or social network website, or a news site, or adictionary site, or mapping service, or a to-do list, or a phone list,or a merchant website, or a shopping website, or a coupon site, or adiscount site, or gasoline price site, or an electric vehicle (EV)charge locator service, or an EV charge reservation service, or ane-payments site, or an energy pricing site, or a route mapping service,or a traffic service or site, or a movie site, or a music site, ortravel site, or a vehicle site, or vehicle manufacturer site, or arental car site, or an airline reservation site, or a restaurant findingsite, or a review site, or a weather site, or a loyalty rewards site, ora database, or a historical driving database, or a vehicle-to-vehicledatabase, or a holiday calendar, or the internet, or combinationsthereof.

This list of exemplary data and services should not be viewed as limitedto the set of data but simply as an example of data can be accessed andprocess to identify contextual related supplemental content 2250. Inoperation 2252, the interaction mode for the vehicle can be identified.The interaction mode can be the mode that's already selected for thevehicle, and therefore the data that that vehicle is expecting will bein accordance with the interaction mode and it settings.

In operation 2254, select supplemental content can be sent to thedisplays of the vehicle or output via the audio system. As noted above,the type of information, such as the select supplemental content istailored for the interaction mode selected for the vehicle, as well asbased on user interactions 2255 and the contextual relationship of thoseinputs to the time, the geo-location, learned preferences, and the like.

In other embodiments, the vehicle may respond to user input or providerecommendations without being prompted by user input. One example of avehicle recommendation may be to inquire if the user wishes a particularsetting. The recommendation may use, for example, a history of use bythe user to determine or predict that the user may indeed wish toimplement such a setting or provide input. In one example, if it is coldoutside, the vehicle may automatically heat the seats to levelspreviously set by the user (e.g., when similar conditions wereexperienced) or provide recommendations to the user. In other examples,the vehicle can automatically seek input from the user with customizeddialogs. By way of example, dialogs may be audio dialogs, text dialogs,icon dialogs, sound and text, voice and text, or simply voice output.One example of voice output may be, “It's cold outside, do you want meto heat the seats?,” or “Hi Fred, its cold outside, do you want moreheat?”, or “Your seats have been set to level 3 heat, etc.” These areonly some examples, of recommendations that can be provided to the user,based on one or more of the user's voice tone, mood, learned priorsettings, use patterns, predictions, and combinations thereof.

In one embodiment, a vehicle system can interface with a user byrequesting simple user feedback, e.g., by providing the user a set ofsimple queries. As vehicle settings become more complex, making settingscan be very distracting, especially when driving. As such, instead ofrequiring a user to navigate screens and settings and inputs, a vehicleinterface system can generate a set of queries, which are then used todetermine a setting or recommended setting. The queries can be, forexample, a set of verbal questions. Once the user answers the questions,the system will narrow down to what the user wishes to do. For example,if the user wishes to check a system setting, e.g., tire pressure, theuser can simply initiate an inquiry. The injury can begin by the user orthe system, and may being by the user saying “tell me the tirepressure.” The system can ask, “which tire are you interested in?, rightfront, left front, rear right, rear left?” The user can say, “leftfront,” the system can say “pressure is 32 PSI, and is low, do you wantto find air nearby?” The user can then say, “yes” and directions areautomatically provided to the vehicle user interface. Thus, instead ofthe user being distracted while driving, attempting to navigate severalscreens to get to vehicle systems, then tires, then pressure, and thennavigate to search for a service station, the vehicle can skipnavigation screens and immediately arrive at the information or setting,which removes distraction to the driver. Further, as vehicle userinterfaces continue to add more functions, users need to memorize newscreens, settings, inputs, etc., and this simple query interchange withthe vehicle will allow instant access to desired settings. The usersjust answer basic questions about preferences, which are thenimplemented as a setting by the vehicle.

In some embodiment, the system can further predict desired settings. Asin the example noted above, the vehicle system predicted that the userwill need to find a service station to get air for a low pressure tire.Prediction can further be refined based on environment conditions,geo-location, time of day, time of week, etc. In one example, thevehicle can determine if you are lost by examining GPS travel patternsof your travels, and other factors (e.g., searching for an address onthe mapping function, making calls, etc.). In one embodiment, thevehicle can verbalize: “you look like you are having trouble findingyour destination, would you like a route to your destination?” In otherexamples, the vehicle can identify that the driver is having troublestaying within the lines of a lane. If the time of day is late at night,and if the user is far from home, and/or the vehicle is traveling to amapped destination, and/or the vehicle has been traveling for anextended period of time, e.g., 6-12 hours, the vehicle system can deductthat the user is getting tired, and may need to rest or find a hotel.The vehicle can say, for example: “Are you ok?” “Are you tired?” “Wouldyou like me find you a hotel or rest stop?” If the vehicle is runninglow on gas, the vehicle may provide a query to the user saying “Chevrongas is available at the next exist, shall I reserve a pump for you?” Ingeneral, the technical aspects of these embodiments is that the vehicleis contextually aware of its surroundings, the user patterns of thevehicle when used by user accounts of users, predicted use patterns,likes and dislikes, setting preferences, general preferences,geo-locations and available services and goods, etc. The vehicle is, inone embodiment able to exchange with the user in simple dialogs, whichreduces the need to navigate touch screens, select icons, enter text,read menus, etc., especially when the user is driving.

In another embodiment, voice interchanges can occur between a user and avehicle, wherein the vehicle is context aware of environmentalconditions (e.g., weather, time of day, day of week, online calendars ofthe user, status of local charging or fueling stations, etc.), andgeo-location of the vehicle, geo-location paths taken, preferred pathsand navigation routes. Additionally, in one embodiment, the vehicleand/or cloud system will maintain or have access to user preferencesthat are associated or maintained for a user account of the user. Thepreferences can include preferred settings of the vehicle, settingspreferred for specific periods of time or conditions, preferred goodsand/or services of the user, preferred routes taken in certaingeo-locations, historical selections made for vehicle settings,biometrics of the user, tendencies of the user, voice profiles fordifferent tones of voice of the user, etc.

In one embodiment, an interactive two-way conversation with thecar/vehicle can be had, instead of commanding the car to do xyz. Whenthe vehicle accesses the user account of the user, the vehicle can makedecisions as to how to respond to the user, and these responses can belearned over time or based on prior settings, such that a confidencescore can be generated for determining how to respond to the user. Inone configuration, since the vehicle is user aware, the user will buildup a relationship with the vehicle. In one embodiment, as is also commonin person-person relationships, relationships between the vehicle anduser can change based on present conditions.

In another embodiment, the vehicle can include a user settable MOODsetting. In one embodiment, the vehicle automatically determines theuser's mood via the user tone, and/or other inputs or voice commands Inanother embodiment, the setting can be manually triggered/set by theuser. For example, a user interface of the vehicle or voice command canbe provided to direct the vehicle to change its mood. Based on the moodsetting, the type of questions posed to the driver can change. Forinstance, if the driver is in a good mood (and the mood=good is set),the car may pose more interesting questions to the user instead of justmaking settings. The vehicle can, suggest a mood lighting of the car, adifferent display panel skin, music, or voice used by the car to talk tothe user. Technically, the personality of the car changes, based onconditions and the associated mood setting. In other embodiments, theuser account will include the age of the driver is, and the personalitycan change over time. In the same manner, if the driver is a teendriver, the personally can change so that the vehicle communicates in amanner that is more pleasant to the teen. The same can be true forsenior drivers, whom may desire more tame personalities and/or settingconditions.

The personality of the vehicle can therefore switch based on who haslogged into the vehicle or who the vehicle has determined to haveentered or is operating the vehicle. The personality of the car can alsochange based on geographic region. If the vehicle is in the southwesternU.S., the car can take on that personality. If the user has an accent,the car can detect a geographic region for the accent, and then uses aregion friendly accent.

In still another example, the car can also predict your language basedon your accent and/or geo-location, and ask you if you wish to interfacein a different language. In this embodiment, the user need not manuallychange the language setting, but the car can change it for the user byanalyzing the language tone, dialect, region, etc. In one embodiment,the system can also determine the driving conditions, environmentalconditions, geo-location, and mood of the user. If the vehicle istraversing a curvy road, in heavy traffic, in bad weather, etc., thesystem may refrain or not ask questions to the user that may distractthe user. However, if the user is at a stop sign or light, the systemmay ask more questions that require more thinking or analysis.

By having the vehicle system be more aware of the user's mood andconditions of driving, the user will not inject verbal interchange withthe user that may cause more distraction. In one example, a goal is tohave the car interact with the user at times that will not place dangerto the concentration needed for driving. If the user is at a stop sign,the vehicle may ask questions that are more complex. The system can alsoselect the type of questions to ask the driver, based on conditions ofday, the mood, or current personality of the user. As noted above, thepersonality of the user can be detected by the vehicle or can be set bythe user. The user can, for example, select a simple button/icon/inputto state the mood. In other embodiments, the mood is determined based onthe voice or tone of the user, and/or biometric data. In one embodiment,the user can answer a set of questions, posed by the vehicle, whichproduces initial settings for the user. In some embodiments, the simplequestions can be asked to train the vehicle, e.g., when the vehicle isnew or if the user has a new user account. In one example, the user thatjust purchases a car can answer simple questions in a questionnaire andthe system will set most of the basic settings to the car. The questionscan be posed by the vehicle itself or can be input via a website ormobile app, which are then transferred to the vehicle. In this manner,when the user drives off the lot in a new vehicle, the vehicle respondsclose to what the user wants. The user can also transfer settings andhis/her answers to questions from prior cars, which may be saved in aprofile of the user (e.g., user account maintained and managed by cloudservices), so when the user buys a new car, the profile is instantly setwith starting settings.

In other embodiments, training of the vehicle may take place, so thevehicle will understand the different tones of voice the user may haveand assign specific voice profiles. For example, the vehicle will learnwhat a happy vs. angry response is, or sad vs. tired, etc. In oneembodiment, the user may be asked to train the voice input system. Thesystem may ask the user to speak some phrases, such as those needed forsimple user interface inputs. For example, without limitation, thevehicle can ask the user to read the phase “Car, turn up the radio”,“Car, warm up my seat”, “Car, give me directions”, “Car, where is thenearest fill station”, “Car, get me home fast”, “Car, find me gas now!”,“Car, find me a restaurant now!”, “Car, when is the next service?”, etc.The system will learn the speech/tone of the user, so that future accessis implemented in a vehicle response that is consistent with the inputand tone. In one embodiment, the predictive response can includecomputing a prediction grid of assumptions, which are strengthen orweakened over time based on use. For example, if the user response asthe vehicle believed the user to have responded, then the nodeassociated with that predictive grid is strengthened. If not, the nodeis either removed or omitted. Further, answering some questions positivewill likely produce a yes on the next node, and so the system can buildup more nodes to the grid, and over time the node-grid connects getsvalidated or not.

In one embodiment, as user drives around, the cloud infrastructure isable to collect and attribute data about where the user is driving(e.g., GPS), access user profiles to get calendar information, scheduleinformation, buying history, etc. In one configuration, contextualdriving enables the system to collect ads (advertisements, discounts,rewards, coupons, loyalty points/credits, etc.) and/or recommendationsfor a later time period when the vehicle is not moving. In anotherconfiguration, the vehicle collects ads and/or recommendations and sendsthem to the user's account so that the information can be viewed later,when the driver is not moving the vehicle. In still another embodiment,the ads may be delivered to the user when the user needs them. Forexample, the driver earns credits over time for some reward or coupon,and the coupon or ad is only surfaced when the user needs it. When theuser arrives at a Coffee Shop, the user's Coffee Shop account will showa discount that the user earned earlier, e.g., by viewing some ad on thestreet, viewing the ad later that was sent to the user's account becausethe user drives by the Coffee Shop often on way to work. As a technicaladvantage, this strategic surfacing of information assists to reducedistraction by throttling when ads and/or recommendations are surfaced.For instance, maybe ads are surfaced only when the use is at a stoplight, only parked, finished driving, before the driver begins a route,etc. Instill another embodiment, the ads may be surfaced to the useronly when the user is alone in the car, or maybe when the user prefersto see or get ads. This is method of throttling is instead of currentmodels where vehicles may be flooded with user interface data, simplybecause the ads are available, instead of when the user wants theads/discounts/coupons or when distracted driving can be reduced.

FIG. 23 illustrates an example of cloud processing by cloud services 120to receive inputs, process user behavior, obtain inputs from vehiclesystems, access historical data, and construction and refinement of auser's data graph 108′. The inputs over time, which are made in thevehicle, are transferred to the cloud server or servers, in terms ofdata updates, and the data is processed by the data graph 108′ processor2306. In this manner, the user's voice input can be processed toidentify tone and/or user behavior and the vehicle response and providesrecommendations and/or settings or responses, based on the learnedinformation obtained from the user data graph 108′. As mentioned above,the user data graph 108′ may be processed by the server alone, or thevehicle alone, or a combination of the server and the vehicle, or acombination of multiple servers and devices. In various embodiments acopy or cache copy is maintained in memory of the vehicle so vehicleelectronics (e.g., specialized hardware and/or software) can applyvehicle response quickly to voice input.

As shown in FIG. 23 , the user A can provide voice input to the vehicle,which uses one or more microphones installed on one or more surfaces ofthe vehicle. For examples, some microphones may be installed near thedriver and some installed in other locations, so as to identify who inthe vehicle is speaking. In some embodiments, sound localization may beused to identify and filter voice input coming from the driver orspecific passengers. In one embodiment, the vehicle may be used by userA, who may be using the vehicle with his or her user profile 2304selected, e.g., via the user account 2302 access. The cloud processing120 can identify the user account and profiles from database 112. Incloud processing, user inputs can be tracked over time, including thevoices profiles used to command or request information from the vehicle.As shown, various systems of the vehicle can be tracked, sensed, and/orlearned over time. This type of processing can include accessingcalendar data, vehicle preferences, vehicle preferences, input patterns,learned preferences, biometrics, conversation inputs, language tones anddialects, driving history, mood patterns, geolocations, and interactionsbetween one or more of these data processing entities.

In one embodiment, the user voice input can be analyzed in operation108, the analysis can be, for example, to identify a tone that the userhas in his or her voice when making a voice command, requesting data viaa voice command, or dictating a text or message for sending by acommunication system of the vehicle 160. As discussed in variousexamples in this application, the tone identification can include takinga sample of audio, processing the audio to identify analogcharacteristics (e.g., utterance sample, magnitude, frequency, durationsbetween peaks in frequency/magnitude, features markers in the sample,lows and highs in the pitch characteristics, etc.). The processing ofthe audio can also include generation of an audio signature, which maybe analog or digital.

In one embodiment, the audio signature is transformed into a digitalsignature, and on or more analog to digital (ADC) operations may beperformed. The ADC processing can be performed by a specialized circuit,integrated circuit, or via software and a processor, or firmware. Inoperation 106, the tone that is identified is mapped to a vehicleresponse. This mapping is, for the specific user, e.g., for user A thataccesses the system via a user account. For example, the cloud services120 can operate to service many connected users, and each user willdevelop its own set of historical data that is usable to define or inferpreferences, tones of voice, user inputs, etc. In operation 109, thevoice profile is identified. The voice profile can be happy, sad, angry,frustrated, hurried, fear, surprise, disgust, etc. Based on the detectedvoice profile, e.g., in operation 109, the vehicle response is made bythe vehicle 160. As noted above, the vehicle electronics may use acached copy of the user data graph 108′, which is used to make decisionsas to the vehicle response to be made based on an identified tone. If adecision is made by the vehicle electronics 161, that input is saved,which may include what the tone was when the user made the input, thematching data between stored tones and the received tone, etc. Thisinformation can be, over time, synced with the server or servers of thecloud services 120.

FIG. 24 illustrates various examples of cloud processing 120, whichincludes servers 2402, storage 2404, and example processing used orinterfaced by user data graph 108′. In one embodiment, a user account2408 is maintained by cloud processing 120, and profiles of the user orsub-users are stored in profile 2409. User data 2410 is also stored, forspecific users. The user data 2410, in one embodiment, may include usersettings 2412 made by the user or recommended to users. Historical userinput 2414 is further used or accessed from a user's data graph 108′.Voice profiles to vehicle response mapping 2416 is also stored orprocessed, along with learning logic 2418. In one embodiment, toneidentifiers (IDs) are used for specific voice profiles. Thus, based onthe user voice input, the tone is identified along with a correspondingvoice profile. Using the voice profile, it is possible to map to avehicle response. Over time, these inputs can be refined using learninglogic, so that the response can change or mold as the user's behaviorchanges, e.g., as the user ages or as the user's day is going, or basedon time of day, based on type of month, or simply because the userchanges his behavior. As noted above, the tone ID can be defined by anaudio signature. The audio signature may be obtained by sampling aportion of audio captured by microphones of a vehicle, which is thenprocessed to produce the tone ID. The tone ID may, in some cases can bedefined by a code, a vector, an analog curve, a digital score that canbe graded, normalized, or categorized. In any of the embodiments, thetone ID, e.g., data that is used to represent the tone ID, can benormalized against the user's historical input, or can be normalizedagainst a larger database on the server using data of one or more users.

FIG. 25 illustrates examples of voice input and vehicle response. Theseare only some examples, and more tone IDs can be constructed, based onthe user's behavior. For example, some users have many types of tones,while some users have minimal changes in tone. Still further, somepeople have wide ranges of personality, while some have very rigidpersonalities that do not change, and thus the tone types can change,increase or decrease for specific user profiles. In the example, theuser voice inputs 2550 can be provided to the vehicle over time, e.g.,in various sessions (e.g., sessions 1-n). In session 1, the user voiceinput 2550 is provided to the vehicle and a voice sample or multiplevoice samples are taken to identify a tone 2552. Using the identifiedtone, tone matching 2554 is performed to identify the type of tone usedto make the voice input. In this example, the tone is determined to be arushed tone 23, in operation 2556 a. The voice profile, for the user A,is determined to be voice profile 32 in operation 2558 a. In operation,using the voice profile 32, a vehicle response 2560 is processed.

In session 2 a vehicle response 2560 is produced for user whenfrustrated. In session 3 a vehicle response 2560 is produced for userwhen happy. In session 4 a vehicle response 2560 is produced for userwhen slow, and so on. Over time, the accuracy of detected tones in auser will be refined, e.g., with more use.

FIG. 26 illustrates an example tone to voice profile mapping function,in accordance with one embodiment. In this example, for a specific userA, based on learning over time, or based on calibration, user's detectedtones will be mapped to specific voice profiles. These mappings may beadjusted from time to time, when it is determined that the mapping isincorrect or is changing. For example, if the user is happy but isproviding loud or forceful voice inputs, this data can be adjusted tomap the tone to a different voice profile. In some embodiments, the toneto voice profile can change depending on the time of day, day of themonth, or when certain events occur to the user or are determined tohave occurred by examination of the user's calendar, etc.

FIG. 27 illustrates example vehicle response, as tied to specifictone-voice profiles, in accordance with one embodiment. As shown, whenthe tone 2 (happy) is identified, the profile 1 is found for the user.Based on the profile 1, the vehicle response is set for the request of,e.g., “show me a map.” In this example, the vehicle may ask if the userwishes to stop for coffee, offer discounts, and can provide a map.Because the user is perceived to be happy, the system makes adetermination that the user may be more receptive to the additionaldata.

For the example when the tone 5 (urgent) is identified, the profile 2 isidentified. Because the user is perceived to be in an urgent state, thevehicle will respond more immediately, supplying the requested map.Since the user is in an urgent state, the vehicle will not provideadditional data, e.g., coupons, discounts, suggestions, etc., since theuser is likely less receptive.

For the example when the tone 6 (sleepy) is identified, the profile N isidentified. Because the user is perceived to be sleepy, the vehicleresponse may offer one or more additional responses. For example, thevehicle may provide a recommendation for a hotel nearby, recommend acoffee shop, recommend a rest area, and provide a map (i.e., that wasrequested). In some embodiments, the vehicle response can includeadditional data, recommendations, or automatic settings. By way ofexample, the vehicle response can also include to automatically changethe temperature in the vehicle to cold or open a window, or turn up theradio, or recommend any of these things or other suggestions. Thus, thevehicle response can be adjusted to cater to the tone of the user, e.g.,so as to provide, augment, modify, moderate, and/or change the vehicleresponse to detected tone in the user's voice.

FIG. 28 illustrates example processing by electronics of the vehicle toprocess voice input and produce a vehicle response, in accordance withone embodiment. In this embodiment, voice input 140 by a user isprovided to a vehicle input, e.g., a microphone of the vehicle. Inoperation 2810 voice sampling is performed. The voice sample can includeidentification of frequency and magnitude markers in operation 2816. Thefrequency markers can include separations between pitches, peaks inmagnitude, waveform characteristics, waveform bursts, energy detected,etc. In some embodiments, it is possible to analyze the voice sampleusing a number of metric in combination, such as waveform analysis,pitch analysis, volume analysis, burst frequency, etc. In oneembodiment, the voice input can be converted to text and the text can beassociated with specific frequency and/or magnitude markers, states,typical inflections, word to word patterns, etc. In operation 2818, alookup is performed to identify a tone that best matches the markers inthe voice samples. In operation 2820, mapping is performed between thetones identified to a voice profile of the user. In operation 2822, thevehicle response is identified and made for the voice profile. The voiceresponse is moderated for the tone identified in the voice input 140.The vehicle computer 2812 can be specialized to handle processing of thevoice input, the voice sampling, and is configured to use wirelesscommunication 2814 to interface wirelessly with cloud processing 120.

FIG. 29 illustrates an example process for receiving voice input,analyzing the voice input to find a tone, and applying a vehicleresponse, in accordance with one embodiment. In one embodiment, thevoice input is received in operation 2910. The voice input is of a userat a vehicle to command input or request information via a voice inputinterface of the vehicle. This voice input is processed by a computer ofa vehicle, e.g., using the user's data graph 108′, which includesinformation on how to identify the user's tones, identify thecorresponding voice profiles, and selection of specific vehicle responsefor the voice input. In operation 2912, the computer of the vehicle isconfigured to analyze the voice input of the user to identify the tonein the voice. The voice input can be, for example, “find me a hotel neardowntown Los Angeles,” “call John Smith”, “find a nearby chargingstation”, “turn on the air”, “heat my seat”, “read me my calendarappointments,” etc. The voice input can therefore be for any command tomake a setting, change a setting, update data, read data, locate data,do an internet search, etc. The voice input 140 is therefore sampled toidentify the tone in the user's voice in operation 2912.

In operation 2914 the voice profile is used to select the one or morevehicle responses. As used herein, a vehicle response can be one or moreactions, and the actions can include to return data, read out data,input data, make a setting, change a setting, send data, read data, makea text, read a text, dictate a message, make a call, recall a map,change the climate, make an entertainment setting, set a securitysetting, learn a setting, without limitations to other examplesdescribed in this document.

FIG. 30 illustrates one example of a vehicle 160 driving compartment andthe passenger seat. In this example, the vehicle will typically includea number of switches, dials, controls, buttons, scrollbars, pushbuttons,levers, which are commonly considered to be physical input devices tocontrol settings or features of the vehicle. Commonly, these physicalinputs will include fixed icons which are typically painted orpermanently engraved on the buttons. For example, the up and down arrowson window opening controls are typically fixed. Other fixed controls mayinclude the buttons for turning up or turning down the air conditioning,buttons for opening a door or locking a door, volume controls for audio,door locks, audio control buttons, seat positioning buttons, and thelike.

In accordance with one embodiment, these physical inputs are configuredto include one or more graphical screens which can change depending onthe interaction mode selected for vehicle. For example, the dial buttonuse for air-conditioning can be changed to a dial button use for theradio. In one further example, a graphical screen can be defined overthe button or physical input or beside or around the button or physicalinputs.

Thus, the user can identify what each button or control is capable ofdoing in the vehicle. In some examples, some of the screens associatedwith physical inputs can be replaced with different icons or controlidentifiers. If the interaction mode is a senior mode, the icons, text,controls, can be magnified in size, type, or can be simplified. In someembodiments, the buttons, icons, text, and associated screens ordisplays for certain ones of the physical inputs can be set inaccordance with the theme associated with the interaction mode, or canbe individually adjusted or changed without regard to interaction mode.FIG. 30 also illustrates example locations of microphones, which may beused to detect voice input, identify who in the vehicle is speaking,mask out nose, and/or focus on specific users (e.g., driver or specificpassengers).

In the illustrated example of FIG. 30 , screens can be positioned indifferent locations to avoid the need to add additional buttonsthroughout the cockpit of a vehicle. In some examples, the steering willcan have screen displays that are easily touched or interfaced with toavoid distraction. Certain screens on the steering wheel may change, forexample to provide access to inputs that would normally be associatedwith physical inputs. In one embodiment, fewer physical inputs areprovided throughout the vehicle and are replaced with configurablescreens that provide the functionality that would otherwise be requiredvia the physical inputs. The example locations on which the inputs canbe provided are simply that, just examples, as they can be providedanywhere in, on or around the vehicle.

This type of interactive display and control provided in vehicles canassist vehicle makers to provide fewer buttons that are physical andreduce the cost and weight of a vehicle. In one example, the steeringwill may have configurable screen to allow the user to adjust thevolume, lock or unlock the phone, change the music, access menu items,access the user's home, ask for help, change the dashboard style, setthe configuration mode, and the like. As further shown, one of theinputs can be to simply toggle between one or more interaction modes.

The interaction mode selected in the example of FIG. 7 is simple. In oneembodiment, the interaction mode can be dynamically switched based onthe user of the vehicle. In one embodiment, the interaction mode can beassociated to a profile that is maintained for a user, and the profilecan be used in different vehicles such that interaction modes can beautomatically applied to different vehicles, whether owned or used orshared or rented. In another embodiment, features of the vehicle can beprovided with sensors, such as cup holder sensors.

In one embodiment, a learning algorithm can be provided to determine ifthe user likes to cool or warm a beverage that may be positioned in thecup holder. For example, if the time of day is morning and the cup inthe cup holder is detected to have warm liquid or a warm temperature,the cup holder can automatically turned on to maintain a substantiallywarm or hot temperature of the contents in the cup (if heating featuresare provided). In another embodiment, an analogous process can beperformed to cool or maintain the cool temperature of liquid in a cupholder, based upon the context of the learning. For example, if it isdetermined to be a hot day, and based on previous patterns of use theuser has selected to keep cops cool in the cup holder's (if coolingfeatures are provided), the system can automatically set the cup holderto maintain the cool temperature of the cup or its contents.

Still further, other sensors in the vehicle, such as presence sensorscan identify whether more passengers or fewer passengers are inside avehicle. Depending on temperature conditions outside, and based onlearning of previous patterns of the user who may have had passengers inthe past, it is possible that the temperature inside the vehicle islowered 3 to 5° cooler than normal. This may be true because morevehicle passengers can raise the temperature of the cockpit, which mayrequire additional cooling. Still further, the context of additionalpassengers can also detect whether additional passengers have their owndevices in the vehicle, and certain settings for those devices can beshared to those devices.

In some embodiments, inside vehicle environmental characterizes may bemeasured with sensors. For instance, an oxygen sensor may be configuredto measure the oxygen characteristics inside the vehicle. Another sensormay be a carbon dioxide sensor, which may be able to measure thepresence of more than one person in the vehicle, e.g., since more peoplewill be exhaling. In some embodiments, temperature sensors in thevehicle can determine when persons are sitting on specific seats. Inother embodiments, heat sensing cameras can identify the presence ofpersons in the vehicle, their seat locations, their body mass (e.g.,adult, child, infant). Further embodiments, may combine outputs from thevarious sensors, which may be integrated with vehicle electronics forprocessing the sensed data, and then use the data to identify thepersons and locations of such persons in the vehicle.

For example, passengers may be able to connect to a Wi-Fi or Internetconnection provided by the vehicle. This connection can identify theother devices are contained or located within the vehicle. Thisinformation can be used to provide those specific devices access tocertain controls of the vehicle. The controls provided can be based onlearning associated with previous privileges granted by the primary useraccount in the vehicle, which is typically the driver. For moreinformation on sharing vehicle controls to user devices, reference maybe made to U.S. application Ser. No. 14/222,670, entitled “Methods andSystems for Providing Access to Specific Vehicle Controls, Functions,Environment and Applications to Guests/Passengers via Personal MobileDevices,” which is incorporated herein by reference.

FIG. 30 further illustrates a hybrid physical/display controls with anassociated interaction mode customization. In this example, a simplifieddashboard of a vehicle is provided. The simplified dashboard can includecertain gauges that are necessary for driving, yet the clutter providedin the vehicle displays is reduced as described above. In this example,there is one physical input to the left of the steering wheel and 3physical inputs to the right of the steering wheel. For simplicity, thephysical inputs are shown as dials.

It should be understood that the physical inputs can take on variousforms such as pushbuttons, toggles, sliders, press in controls press outcontrols pull out controls twist controls shift controls etc. Continuingwith the example, for purposes of understanding, the four physical inputdials may be allowed to twist turn be pressed be pulled or selected. Inaccordance with one embodiment, the physical inputs may also include ascreen that provides the readout of the text shown in the examples. Thistext can be configured to change based on the interaction mode selectedfor the vehicle. In one embodiment, a senior mode can be selected forthe vehicle, which may indicate that the dials should be maintained assimplified as possible and any text should be amplified or enlarged sothat control and access is easier for older adults.

For example, one button may simply indicate wipers, the next buttonvolume, the next button tuning (audio), air conditioning, and the like.As noted above, more than four physical buttons provided with a hybriddisplay can be provided, and this is only one example to convey thesimplicity of configuring displays associated with physical inputs. Inanother example, an intelligent mode may be selected for the vehicle,which may dynamically change what is displayed on the physical inputs.The display the physical inputs can be a small LCD screen, atouchscreen, proximity non-touch screen, gesture input screens, icons,text, combos of text and icons, colors, a screen typically used on asmart phone, or the like.

Thus, users can provide touch input to the buttons similar to the waytouch input is provided on screens of a smart phone or a tablet. Inaddition, the buttons can also be manipulated as physical inputs toprovide an additional level of change, input or interaction. In anotherexample, the interaction mode can be changed to provide for an informedmode. The informed mode can also again change was displayed on the facesor surfaces of the physical inputs.

Thus, the physical control devices may be pressed, tuned, dialed,touched on the display screen parts etc. In still another embodiment,the display screens can be provided beside the physical inputs so thatthe screens are not on the buttons themselves. For example, the surfacebeside underneath or above or below the physical inputs can includesmall screen that dynamically changes to identify what the physicalinputs are provided to do. Thus, based on the configuration provided tothe physical inputs, the content displayed on the physical inputs orbeside or around the physical inputs can define what functionality thoseparticular inputs can render for the vehicle. As such, the dynamicadjustments and changes to the physical inputs of a vehicle can furtherassist in customizing the users feel and interaction with a particularvehicle.

This functionality provides a further dimension in customization forvehicles, at certain drivers prefer less technology while others prefermore technology, and others prefer simplified interfaces common in oldervehicles. These customizations provide for less distraction to driversand should improve safety, as the interfaces are provided in a way thatis most comfortable to the specific user.

FIG. 31 illustrates an example of a car dashboard having interactionstyles and screens and settings therefore, in accordance with oneembodiment. In this example, the text and icons and language can alsochange for specific vehicle based on the configuration settings,interaction modes, and customize settings by particular users. In theseexamples, one interaction style can be a factory setting. The factorysetting can provide definitions for adjustments that can be made toclimate events, seat controls, windows, latches, and the like.

FIG. 32 illustrates example vehicle interaction settings and styles.These settings can be made via any number of interfaces, e.g., inputscreens of a vehicle, voice inputs, portable device inputs, etc. In someembodiments, voice inputs can be made to set any one of these settings,making changes, updates, etc. In some embodiments, the inputs can alsobe via one or more touch inputs, gesture inputs, etc. The modes can, inone embodiment, assist in automatically changing groups of inputsettings and/or interfaces in the vehicle. In some embodiments, thevoice input can be treated differently depending on the user making theinput and/or based on the selected mode. For example, if the mode is“senior mode,” the voice profiles can be used for senior mode, andassociated mapped vehicle responses can be made. The same can be done,e.g., if the user is a teen, and the mode is “fun mode,” and othermodes.

Furthermore, a senior mode can be provided with icons, text, languagefamiliar to a specific user or demographic. Another example mode may bea fun mode, which allows more customization to certain features. Thiscustomization can be very complex and can also be descriptive instead ofin the form of express settings. The descriptive inputs can betranslated to specific inputs by the computer. It should be understoodthat these particular settings can be predefined by the user either inthe vehicle, via a user device, at a website, or some other way that canbe associate to a user account and a given profile of that account.

Learning can therefore take place over time for that specific profile.The user account may also be transferred from vehicle to vehicle if thevehicle supports application of interaction modes, or application ofcertain settings saved in a profile of the user account. It should beunderstood that some vehicles may not have interaction mode capability,but may allow customization of certain features automatically based onthe user account.

This customization may allow for at least one or more features to beautomatically transferred from one vehicle to another. In other moreadvanced vehicles, more configuration modes are settings can betransferred or used in specific vehicles. Accordingly, the amount ofcustomization and dashboard transferability from vehicle to vehicle willvary from few to more features being configured or transferable,depending on the manufacturer, the software, the systems integratedtherein, and the functionalities of the vehicle.

FIG. 33 illustrates one example of voice input processing 3310. Asmentioned earlier, the vehicle processing can be performed onelectronics of the vehicle, e.g., using the data graph 108′. The datagraph 108′ may be a cached copy of the user data graph 108′ maintainedin cloud processing 120 for the user. In processing by the vehicleprocessing 3310, the voice input 140 is captured by one or moremicrophones 3320 of the vehicle. One or more portions of memory, e.g.,RAM, DRAM, solid state drive (SSD) cache, or hard drive storage, areused to store or cache some of the voice input. In one embodiment,amount of voice input can be one or several seconds, which may be enoughto identify some tone or markers in the tone of voice of the user. Theaudio cache 3322 is used to hold enough analog voice sounds to enableanalysis. In some embodiments, the audio cache 3322 can store multiplesamples, e.g., several words, statements, voices statements made overone or more sessions, days, months, years, etc. In one embodiment, noisefilters 3324 may be implemented in the form of analog and/or digitalfilters to remove certain frequencies, sounds, background static, etc.In some examples, the noise can be other people talking in the vehicle,road noise, wind noise, entertainment noise, and combinations thereof.

Then, a portion of the audio that is in the audio cache 3322, e.g.,which may be before or after noise filtering, can be audio sampled 3326.The audio sampling, in one embodiment, can include performing frequencyanalysis, magnitude analysis, modulations, tone shift analysis, markeridentification in the sampled audio, peak analysis, intensity analysis,duration analysis, modulation changes, etc. The audio sampling can occurin the analog domain or in the digital domain, or part in analog andpart in digital.

In one configuration, based on the analysis, an audio signature 3328 isproduced. The audio signature may be for one audio sample or the resultof analyzing two or more audio samples. For example, one audio samplemay be for one word, a group of words, a sentence, a period of time,e.g., 1 second, 3 seconds, several seconds, etc. In the example shown inaudio sampling 3326, the various wave forms may be for specific words,such as “show me a map”, wherein 3326 a is for “show,” 3326 b is for“me”, 3326 c is for “a”, and 3326 d is for “map.” In this example, theaudio samples show that the user emphasized the words “show me” and thenbe emphasized the words “a map.” This may be consistent with the user inan urgent state, who wishes to get a map urgently. This can bedetermined based on prior user voice analysis, and validated based onthe user's response to the vehicle responses actions when voice input isused. Over time, the determination of which audio signature matches theappropriate vehicle response can be updated. As mentioned above, theusers data graph 108′ can be updated from time to time, based on thetype of response the user gives to the vehicle response. If the vehicleis responding appropriately, then the vehicle response can be reinforcedand applied additional weighting. If the vehicle response is notresponding appropriately, a downgraded weighting factor can be appliedto influence correction of the vehicle response for future inputs.

The audio signature is, in one embodiment, representative of thecharacteristics of the user's voice, when making the voice input(s) 140to the vehicle. In one embodiment, the audio signature represents theidentified tone in the user's voice. The audio signature can thereforebe used to perform matching 3330. The matching is to the voice profiles3336 of the user, e.g., as defined or accessed from the user's datagraph processing 3334. In one embodiment, the audio signature may beused to identify voice profile 72 for a user A. The voice profile 72 isthen used to select or map to a type of vehicle response. In thisexample, the voice response is type B, but a specific type is B2. Forinstance, the voice input may be, “find me a hotel.” The response B maybe a map to a hotel. The response B may be B2, finds a map to a hoteland finds a discount, if the user has a tone that would be receptive togetting discounts. If the user is angry, the user may not be receptiveto hearing about discounts, and a simple address is returned a localhotel. In a sense, the response by the vehicle is moderated for the tonein the user's voice. As shown, a vehicle system function can set. Thevehicle function can be, for example, application of brakesautomatically to avoid an accident, speed controls, auto-drive modes,speaker settings, user interface settings (e.g., providing maps, hotels,deals, locate data, voice-type text messages, etc.). The display may beadjusted, and settings may be made to any controllable part of thevehicle.

FIG. 34 illustrates an example of using touch input to characterize thetype of vehicle response. For example, based on the touch inputcharacteristic, the vehicle response can be moderated. In operation3402, the vehicle response is received. In operation 3404, the touchinput is examined against user data graph of learned behavior andcharacteristics of touch. In operation 3406, a touch profile isidentified for the touch input. For example, the touch inputcharacteristics can be hard contact, hard presses, repeated touches withdifferent force, gesture speed, hand motion rates, softness of inputtouch, contacts by the user on other surfaces, e.g., griping wheel hard,soft, repeated squeezing, etc. In operation 3408 the vehicle response isidentified for the touch profile. The vehicle responses can be, forexample, similar to those made in response to voice input. In someembodiments, the vehicle response can different or in addition to thosemade in response to voice inputs. In operation 3410, the vehicleresponse is applied by electronics of the vehicle, e.g., viacommunication to specific vehicle systems direct, via applicationprogramming interfaces (APIs), or via combination of electronic inputs,mechanical inputs to mechanical components of the vehicle orcombinations thereof.

FIG. 35 illustrates an example similar to FIG. 23 , except that theseinputs relate to touch inputs. In this example, user touch analysis canbe performed in 3502 in order to identify the type of touch. Forexample, the touch can be hard, soft, slow, repeated, intermittent, canhave specific durations, specific accuracy, etc. In operation 3504, thetouch characteristics are mapped to specific vehicle response(s). Inoperation 3506, the touch profiles are identified for the touch input,similar to the processing done for voice input. In these operations, theuser data graph 108′ can be accessed to determine what specific touchprofiles mean for specific users, e.g., using one or more of thecollected historical input data. For example, the touch can analyzed todetermine what types of input mean for identifying the user's mentalstate, mood, and/or behavior.

FIG. 36 illustrates an example of using touch input to identify a user'smood and then to identify a vehicle response. In one embodiment, touchcharacteristics can be arranged or quantified by touch characteristicsIDs 3602, based on the received and analyzed touch. The touch profilecan be identified, e.g., from the user's data graph, which includeslearned data regarding touch input by the user. Mapping 3604 can then bedone, such as to identify touch characteristic IDs, and then a vehicleresponse can be made by the vehicle 160.

FIG. 37 illustrates several examples of using touch characteristics (TC)in order to determine the mood, state, or mental state of the user whiledriving. Generally, touch characteristics can be monitored over time,for example during multiple sessions of use by a user. The more the useruses the vehicle and the touch characteristics are sensed and processed,the users data is graph will continue to be reinforced an optimized toensure that the proper reading of the user's mood, state of mind, ormental state can be detected so that the proper vehicle response can bereturned and applied by electronics of the vehicle 161. In someembodiments, the touch characteristics can include sensing whether touchaccuracy has low, medium, or high accuracy. Touch characteristics canalso include determining a touch duration, which may include short,long, medium, or any value in between. Touch characteristics can alsoinclude touch intensity, which can include low touch intensity, mediumtouch intensity, and high touch intensity. Generally speaking, the touchcharacteristics are measured using a touch surface on the vehicle, suchas an input screen, a button, a surface, a consul, a steering wheel, awindow, a glass surface, a touchscreen, a joystick, a gear shift, aglove compartment, a handle, a toggle button, a switch, an object, orany other service or thing that can be touched and interfaced with.

Generally speaking, the touch surface can be a surface or object the canread or sentence using sensors integrated with the vehicle to identifythe type of touch profile being provided to the vehicle. As illustratedin FIG. 37 , the touch input 3750 can be received by some interface ofthe vehicle as mentioned above. The touch input is then provided totouch characteristic analyzer 3752, which determines the type of touchcharacteristic generated by the users input or control. For instance,the value may include some number for touch accuracy, touch duration,touch intensity, and these values can be matched by a touchcharacteristic matching module 3754. The matching attempt the match frompreviously saved models which type of touch input was received by theuser. Based on the matching, operation 3756 a determined that the touchprofile was a rushed profile, which corresponds to touch profile 32 inoperation 3758 a given the profile 32, the vehicle response 3760 cantherefore apply the input to the vehicle electronics 161. The vehicleresponse, as mentioned above, can include making a setting input to thevehicle that causes the vehicle to make a change to a physical object(e.g. seat, temperature, brakes, steering wheel, window, entertainmentsettings, Internet settings, automatic driving parameter, cruisecontrol, etc.).

The vehicle response can also include returning information back to theuser, such as information requested from the Internet. Information thatis returned as a vehicle response can also be moderated, or changed inform so as to reduce distractive driving. As illustrated, the processcan continue over time in various sessions, such as sessions 2, 3, 4. Insession 2, the touch input was detected to identify a frustrated user inoperation 3756 b. In session 3, the touch input characteristics wereprocessed to identify a happy user in operation 3756 c. In session 4,the touch input characteristics were processed to identify a mad user inoperation 3756 d. In various embodiments, the touch characteristics canbe blended and analyzed in conjunction with voice input. For instance,the touch input of being rushed can be contrasted with the voice inputwhich may have detected that the user is frustrated. In someembodiments, a weighting scheme can be processed to identify whether tocategorize the touch input and voice input as a rushed or frustratedinput. If the rushed touch input scores higher than a frustrated voiceinput, then the rushed mood will be identified and used to identify avehicle response 3760. Other examples of processing touchcharacteristics are described in this application, and the examplesprovided with reference to FIG. 37 should not be limiting to the variousidentified mental states or user states or mood states that can beidentified by way of touch characteristic analysis.

FIG. 38 illustrates an example of processing operations that can be usedto map touch characteristics to touch profiles. It should be understoodthat the mapping would be specific to a user, such as user A. Each userthat utilizes the system will over time build a user data graph thatwill be optimized based on learned input settings, which enable moreaccurate mapping of touch input to specific touch profiles. Over time,certain touch inputs can be mapped to a set of touch profiles, whichenable the system to generate a vehicle response that is most specificto that touch input characteristic. As an example only, certain touchinputs can be mapped to touch characteristics (TCs), and the touchcharacteristics can be assigned specific touch profiles. For example, ifit is determined that the touch input is of a sleepy user, which may beproviding touch accuracy with very low levels, very short durations, andvery light intensities, the touch profile 12 will be utilized forgenerating a vehicle response.

The vehicle response can be to provide information requested for thattouch input. In one embodiment, in addition to providing the liberalresponse to the touch input, the system can also provide additionalinformation or recommendations based on the detected mood of the user orstate of the user. For example, if the user sleepy, in addition tochanging the radio station, the system may also recommend to lower thetemperature of the vehicle, or may recommend a hotel if the user is farfrom home, or may recommend more elevated music, or may reduce thetemperature of the seat, or may request that the user pullover and rest,or other information that would be beneficial to the user is detectedphysical or perceived mental state.

On the other hand, if the user's detected to be frustrated, which can bedetected by detecting low accuracy for the touch input, a shortduration, and a high-intensity, a touch profile 6 can be identified. Thetouch profile 6 may dictate that the users input be immediately shown tothe user or set by a vehicle system. This immediate application of avehicle response may avoid providing the user with additional extraneousinformation, recommendations, discounts, or peripheral information dueto the users frustrated state of mind that is perceived. As such, thevehicle response is moderated based on the detected perceived mentalstate of mind, users mood, and generally the processed informationregarding the type of touch input provided to some object or thing ofthe vehicle, while the users driving. It is again mentioned that thisprocessing of touch characteristics can be in addition to the processingperformed for voice input (if voice input is also provided).

FIG. 39 illustrates an example of various touch inputs that identifytouch characteristics of happy, urgent, and sleepy. As mentioned above,the touch characteristics can be identified based on several variablesof analysis, such as accuracy level, duration level, intensity level,and other metrics. Once the touch profile has been identified, thevehicle response is processed. Similar to the operations performed inFIG. 27 as it relates to voice input, the vehicle response can bemoderated based on the detected touch profile. Again, if the vehicleuser is detected to be sleepy and has a touch characteristic 6, which ismapped to sleepy, the touch profile will allow the vehicle to provide avehicle response that includes additional information. This additionalinformation may be recommendations to avoid accidents, reducedistractive driving, and improve the overall safety of the driver.

FIG. 40 illustrates an example of a vehicle performing analysis of touchinput 4010, and providing a vehicle response 2824. As shown, the vehicleis in communication with cloud processing 120 via wireless communication2814. The vehicle computer 2012, in one embodiment, communicates withother computing resources of the vehicle, such as microprocessors,digital signal processors, application specific integrated circuits(ASICs), mechanical systems, motors, communications systems, and otheroperational systems that can take input and supply an output. In oneembodiment, touch sampling 4012 can be performed by the vehicle computer2012. Touch sampling can include identifying accuracy, duration,intensity, and position markers in operation 4014.

This processing will identify the type of input that the user may beproviding with a finger, a hand, a gesture, or an object that isinterfaced with a surface, or object of the vehicle. In operation 4016,a touch characteristic is identified the best fits and matches themarkers in the touch sample. Based on the touch characteristics inoperation 4018, a touch profile is identified. In operation 4020, thevehicle response is identified to be made for the touch profile that wasidentified. And then in operation 2824, the vehicle system or input tothe vehicle or response to the user is performed based on the identifiedtouch input. Generally speaking, the touch input is designed to replywith the intended function of the touch input, and the reply can bemoderated based on the detected touch input characteristics.

FIG. 41 illustrates an example of method operations that can beperformed in order to utilize touch input to identify a type of vehicleresponse to be performed by the vehicle. In one embodiment, the touchinput is received at the vehicle in operation 4102. In this operation,the input is for commanding an input or requesting information from atouch input interface of the vehicle as processed by a computer of thevehicle. In operation 4104, the touch input is analyzed to identifytouch characteristics of the touch input. Examples of this analysis havebeen described above and in other portions of this application.Generally, the analysis will identify or attempt to characterize themood or perceived mental state of the user, so as to modify or moderatethe response performed by the vehicle in response to the touch input.

In operation 4106, the touch profile is identified for the user for theidentified touch characteristic. The touch profile is used to select avehicle response that is moderated for the touch characteristics of thetouch input. In one embodiment, a database is used to map or containmappings that can be used to as mentioned above, and this processing caninclude utilizing a learning algorithm that can identify the types ofinputs and vehicle responses that best suits of the vehicle touchinputs. Over time, the accuracy of the touch input and vehicle responseis improved, by adjusting the vehicle responses based on actual receiveduser replies or actions based on the provided inputs or recommendationsthat are generated.

FIG. 42 illustrates an example of vehicle processing that can beperformed to identify passive ambient conversational inputs 4210, inaccordance with one embodiment. In this example, a driver may be havinga passive conversation with one or more passengers in the vehicle. Ifthe vehicle is set to a passive audio sensing mode, the microphones ofthe vehicle can be utilized to listen to passive conversations occurringin the vehicle. As mentioned above, one or more microphones may bedistributed or integrated on different surfaces within the vehicle. Themicrophones can isolate and locate the specific users in the vehicle, byway of where they are seated and the location from which voice isemanating. This allows the system to identify when specific passengersare speaking, such as the passenger seat, the rear left seat, the rearright seat, the rear middle seat, or the driver himself or herself.

In this passive mode, the conversational audio input can be analyzed byconversational sampling 4212, which can be executed by the vehiclecomputer 2812, and other computing resources within the vehicleelectronics. In some embodiments, the vehicle electronics can includeaudio sampling circuits that can convert audio conversations to text,and filtering circuits to remove ambient noise or other noise associatedwith the vehicle. In some embodiments, filtering circuits can be used tofilter out specific conversations or conversations that are not centralto the main conversation. For instance, if the drivers having aconversation with the passenger, a 3rd conversation between tworear-seat passengers can be filtered out, as being less important. Inother embodiments, the rear conversation can be considered moreimportant than the conversation between the passenger and the driver. Instill other embodiments, multiple conversations can be analyzed toidentify which conversation is providing more contextual informationthat may be relevant to the current use of the vehicle.

In one embodiment, the conversational sampling performed by operation4212, is performed with the assistance of operation 4216 which canidentify keywords, the tones, the needs, the anticipation of specificrequirements, and markers in the conversations. For example, if it islunchtime and the passengers are discussing a place to eat, the vehiclecan provide recommendations based on the contextual discussion occurringin the vehicle. In one embodiment, in operation 4218, the keywords thatare found are used to identify data relevant to the conversation. Inoperation 4222, the system will identify a vehicle response to be madefor the conversation. In operation 4224, the vehicle system or input tothe vehicle or response to ambient conversation is processed. Forexample, if the users were discussing a place to go out to lunch, thevehicle system may present options on the display screen that may berelevant to the conversation that is occurring. In one embodiment, theconversation that is being passively listened to by the system can beonly temporarily cached and automatically deleted. In one embodiment,the system will allow the user to identify a period of time during whichthe conversation data will be saved, such as cached in history. In someembodiments, the system can be set so that no saving occurs at anypassive conversations, and the data is simply used to provide real-timerecommendations to the user.

FIG. 43 illustrates an example of using conversation keywords (CKW) toidentify the types of vehicle responses that will be provided to theuser. For example, if the conversation keywords included the wordshungry, restaurant, burgers, then the primary keywords can be identifiedas hungry. The conversation profile can then be used to set a vehicleresponse. The vehicle response can be having the vehicle ask the user ifthey wish to stop for burgers, offer discounts, and then provide a map.In the same manner, if in the conversation the keywords are hurry,hospital, then the conversation sample will provide and display a map tothe hospital immediately. Similarly, if the conversation keywords areyawn, tired, bed, then the vehicle response can provide certainrecommendations. The recommendations can be to provide informationregarding hotels, coffee shops, rest areas, and maps. In someembodiments, the vehicle response can also provide additional settingsare information, such as changing the temperature the vehicle to cold,opening the window, turning up the radio, or recommending that the userperform any of these functions due to the detected sleeping nature ofthe vehicle driver.

In one embodiment, at a remote location, a user is able to access a userinterface for an application, which provides users access to useraccounts. A user account can be for a user and the user can add one ormore vehicles, objects, data or appliances for remote reporting, viewingand control. In one embodiment, a user is an owner or user of a vehicle.The user can register the vehicle with a remote service.

The remote service can be accessed over the Internet, such as via awebsite or application of a portable device. The remote service canprovide a multitude of cloud services for the user, such as remotecontrol features, remote viewing services, remote alarm controls, remotecamera activation, remote audio/video recording of the vehicle (i.e.,areas around the vehicle and inside the vehicle). In one embodiment, thevehicle is able to connect to the Internet (e.g., when the vehicleengine is off, on, and/or is occupied or un-occupied) to allow a user,via a remote cloud service, to access features of the vehicle. Thevehicle can be accessed when running, when parked, when stopped, whenmoving, etc. The vehicle and its audio recording devices and videocameras can be accessed from remote locations, to allow users toremotely communicate with the vehicle or with people riding or residinginside the vehicle.

The remote communication can also allow a person to communicate remotelywith people standing outside (or inside) of a vehicle. For instance, ifa user is accessing his or her vehicle from a remote location, camerasinstalled in and/or on the vehicle allow the remote user to see a personstanding proximate to the vehicle. The remote user can then communicatewith a person standing proximate to the vehicle using microphones andspeakers of the vehicle.

In some embodiments described herein, vehicles, structures and objectsmay include circuitry and communication logic to enable communicationwith a cloud processing system over the Internet.

In one embodiment, the services provided by the electronic systems of avehicle can include services that access the various components orsubsystems of a vehicle, such as door locks, service histories, userprofiles, audio settings, entertainment settings, mapping functions,communications systems, telecommunication synchronization systems,speakers, heating and cooling functions, auto-engine start/shut-offremotely via smart devices, remote heating/cooling initiation, remoteface-to-face conferencing, etc. The electronic systems within a vehiclecan also provide a user interface, such as a graphical user interface.The graphical user interface can include a plurality of buttons,controls and transceivers to receive input from a user. The input from auser can also be provided by voice input, facial recognition, eye-retinascans, fingerprint scans, a combination of biometrics, or via acapacitive or regular touchscreen contained or displayed within thevehicle, the vehicle's glass, doors, dashboard, etc.

In one embodiment, accuracy of input, whether it be voice, touch, touchand voice, etc., may be measured. By way of example, if the input istouch on an input screen, surface, glass, monitor, display, etc., (e.g.,by a finger or fingers of a user), accuracy of the input can bequantified. In one embodiment, a determination can be made if the touchcoordinates (e.g., finger contact area(s)) are no more than 10% awayfrom icon. In another embodiment, a determination can be made if thetouch coordinates are between 10% and 25% away from icon, or touchcoordinates are greater than 25% away from icon. The closer the touchcoordinates are, the accuracy can be calculated to higher levels ofcertainty.

In another embodiment, touch duration of an input touch by one or morefingers can be measured. For example, measurements can be made todetermine if the touch duration is short, e.g., less than 1 second,average, e.g., about 1-1.5 seconds, or long, e.g., more than 1.5seconds. Measurement of time duration of an input can be characterizedto determine a person's mood and/or urgency. For example, if the user isrushed, the user may press on a selection quickly, whereas persons thatmay be relaxed may press on selections, icons, buttons, surfaces, etc.,with more duration or latency.

In still another embodiment, touch intensity (using pressure sensors orcapacitive area pressed), may be used. These sensors, which may beintegrated with the input surface, display surface, glass surface,fabric surface, or other vehicle input surface, can be configured tosense an intensity of pressure. By way of example, certain forces may bepredefined to be light pressure, average pressure, hard pressure, etc.Calibration by users may determine what type of an input is, based on auser profile. For instance, the user profile for a large man mayidentify a pressure of 7, between 1 (low pressure)-10 (very highpressure), to be a light pressure. For a smaller person, a pressure of 2can be determined to be a light pressure. In the same manner, otherlevels of pressure between very soft and very hard can be determinedbased on the individual user and data that can be saved to the userprofile, for example. In some embodiments, pressures by particular usescan be learned, to identify a pattern of what it means to be soft, hard,medium, etc., and this learned data can be saved to a user's profile. Asnoted above, the updates to a user's profile, which may be learned overtime, may be updated to a user account of the user on cloud processingservers.

In yet another embodiment, a hybrid approach for determining types ofinput can be processed. By way of example, some touch inputs that occurwhen voice inputs are made can be matched or correlated to identify aperson's mood. Based on the person's mood, which in one embodiment canbe correlated using multiple sensed inputs (e.g., voice, touch, touchpressure, touch duration, and combinations thereof), the vehicleresponse can be moderated to fit or correspond to how the user wishesthe vehicle to respond. As discussed in various examples in thisapplication, the vehicle response is configured to learn what isappropriate to a particular user, based on how the user interacts withthe vehicle. For instance, if the user is looking for fuel, and the useris stressed, the user is likely to want to hear about promotions. If theuser appears sleepy, the vehicle response should act to assist thedriver to be more awake, e.g., such as automatically changing theinternal cabin temperature to a colder state, lower a window, raise avolume on an entertainment system, and even suggest a stop or find alocal hotel if the user is believed to be far from home. In some cases,a touch profile alone can be used in a similar manner as a voiceprofile, to identify a user's mood, and thus adjust or modify a vehicleresponse that best matches the user. Based on the user's response to thevehicle response or a number of vehicle responses over time, the vehicleresponse can be adjusted further based on learned inputs or response bythe user. The learning thus allows the vehicle response to be refinedand further customized to the user. In one embodiment, the profile ofthe user, which is part of the user account managed by the cloudprocessing, will continually update settings and responses, so that amore custom and accurate response can be provided to each user.

In one embodiment, users are also provided with an ability to manuallycalibrate or train a vehicle system, and settings, which are managed bythe vehicle system and the cloud processing system. In one embodiment,the vehicle 160 can ask a user to provide voice samples, touch samples,input samples, etc., to qualify or train the electronics of the vehicleregarding one or more moods of the user. For instance, the vehicleelectronics can be programmed to run a calibration for each user, suchthat each user is asked to repeat several phrases and to repeat themusing an angry tone, a rushed tone, a happy tone, an annoyed tone, arelaxed tone, etc. Based on these responses, the vehicle electronics cananalyze the input to generate audio signatures for the user thatrepresent tone identifiers. The tone identifiers are associated withvoice profiles that can be initially mapped to particular vehicleresponses, based on the known or determined or best fit mood found fromlater voice input. Over time, the vehicle electronics and/or the cloudprocessing system can refine the mappings, based on actual use. Forexample, if the user did not provide an accurate enough sample of beingangry, later inputs with an angry tone can be trained. In one example,responses by a user to a vehicle response, can determine if the vehicleindeed identified the correct mood. If the wrong mood was identified,the vehicle and/or the cloud processing can refine the mappings.

In some embodiments, the system can be set by settings or input toignore certain voice inputs. For instance, the user may be provided withpreferences that would allow a user to choose what functions shouldreact to tone and which not. For instance, react to tone for alteringvisuals on dash but ignore tone for showing restaurants.

In still another embodiment, the vehicle may be set to operate invarious modes. One mode, e.g., a passive mode, may be to allow the userto listen to passive voice discussions in a vehicle. These voicesdiscussions may be saved locally for a short duration to enableprocessing and then discarded. The vehicle may have an input setting toclear all past voice discussions or history of voice discussions. Insome embodiments, passive mode may be disabled. The passive mode may,for example, enable the vehicle to listen to conversations happening inthe vehicle. The vehicle can parse the discussions to identifysemantics, context, and or specific words or phrases. In one particularexample, two people in the vehicle may be discussion where they want togo eat for lunch, e.g., “where do you want to eat? I don't care, wheredo you want to eat? I don't know, just pick something!!, ok steak.” Ifthe vehicle is operating in passive mode, a portion of the displayscreen may be configured to automatically find nearby steak houses, findratings, get driving directions, etc.

In one embodiment, vehicles can maintain information regarding wherethey are, where they are heading and their destination maintained whichis maintained by GPS and navigation systems on board. The informationcollected and maintained by every vehicle may be mutually exclusive,meaning that each individual vehicle is aware of its own heading, rateof speed and current location. This information, in one embodiment iscrowd sourced and/or crowd shared/consumed for use in for accidentavoidance or other communication. By networking vehicles within acertain radius together, all individually location-aware vehicles becomeaware of all other vehicles in their sphere of influence. Vehicles maynetwork with vehicles in their range using wireless communicationsystems such as but not limited to Wi-Fi, Wi-Gig LTE, cellular, radio,near field communication or other methods.

In one embodiment, the communications of the vehicle and electronics ofthe vehicle will enable direct communication with a user of the vehicle.The user of the vehicle can include, for instance, the owner of thevehicle, a driver of the vehicle, or any third party having access tothe vehicle (either to drive the vehicle, to monitor the vehicleremotely, etc.)

The access to the data can also be encrypted to prevent unauthorizedaccess to the data. GPS and mapping services can also be incommunication with the cloud processing 120 provide data concerning thelocations of the vehicles and activities that occurred to the vehicleswhen at particular locations. The cloud processing 120 can be access bythe vehicles themselves using their electronics and communications, viamobile devices, from home, from work, etc.

In some embodiments, the vehicles may establish peer-to-peer links tofacilitate fast transfer of data. In other embodiments, vehicles maylink to each other using pairing algorithms that allow the vehicles toexchange data using WiFi, Bluetooth, near field communication (NFC), orsome other short range communication protocol.

A user's APP homepage may also include dynamically updating sections inwhich the most relevant information at a given time may be displayed orsurfaced to a user. If a user has parked in a certain parking area, heor she may want to monitor metrics related to incidents that may haveoccurred to his or her vehicle, vehicles around his or her vehicle, anydynamically received alerts, as well as precaution levels. Additionally,a user may choose to configure his or her APP homepage to display themost pertinent audio and video feeds to their needs.

In one embodiment, the vehicles can communicate directly with each othervia a temporary pairing process. The temporary pairing process can beautomatically enabled when vehicles become too close to each other, forexample. When this happens, local communication between the vehicles,such as a peer-to-peer connection, Wi-Fi connection, NFC connection, orBluetooth connection can be established to enable the vehicles to shareinformation concerning their proximity to one another.

This local communication will enable one or both vehicles to takecorrection actions or alert a driver to change course or triggerautomatic collision prevention measures (e.g., more aggressivenotifications to one or both operators, slow the speed of one or morevehicles, change the driving direction of one or more vehicles, etc.).Once the close proximity communication occurs and some corrective actionis made, the data regarding the occurrence and the actions taken can becommunicated to the cloud system for storage. The information can thenbe viewed by a registered user having access to an account for thevehicle(s).

The various embodiments may be embodied in computer readable media,which is saved in storage. The storage may be saved on cloud storage,data centers, or the like, which are accessible over the Internet. Theaccess may be wired or wireless. In vehicles, the connection to theInternet may be wireless, and the connection can be continuous ornon-continuous depending connection. Code on the vehicle electrons canexecute at least some of the method operations when not connected andother operations are executed jointly between vehicle electronics 161(e.g., memory, code and processors of a vehicle) and cloud processing,which may implement one or more servers, either virtual or not.

It will be obvious, however, to one skilled in the art, that the presentinvention may be practiced without some or all of these specificdetails. In other instances, well known process operations have not beendescribed in detail in order not to unnecessarily obscure the presentinvention. Some embodiments are defined by combining features fromembodiments defined throughout the present application and materialsincorporated by reference.

In some implementations, the learning and predicting embodiments mayutilize learning and prediction algorithms that are used in machinelearning. In one embodiment, certain algorithms may look to patterns ofinput, inputs to certain user interfaces, inputs that can be identifiedto biometric patterns, inputs for neural network processing, inputs formachine learning (e.g., identifying relationships between inputs, andfiltering based on geo-location and/or vehicle state, in real-time),logic for identifying or recommending a result or a next input, a nextscreen, a suggested input, suggested data that would be relevant for aparticular time, geo-location, state of a vehicle, and/or combinationsthereof. In one embodiment, use of machine learning enables the vehicleto learn what is needed by the user, at a particular time, in view ofone or more operating/status state of the vehicle, in view of one ormore state of one or more sensors of the vehicle. Thus, one or moreinputs or data presented to the user may be provided without an explicitinput, request or programming by a user at that time. In one embodiment,reference is made to learning and prediction, wherein both terms may bereferencing the same or similar function, e.g., looking at userinteractions, preferences, tendencies, etc., in order to identify orselect a particular type of data that may be useful for the user basedon the learning or prediction. In other embodiments, learning may bedefined closer to the traditional sense of machine learning, patternlearning, historical data input analysis, etc., while prediction is maybe defined closer to the traditional sense of identifying some data,which is predicted to be relevant based on analysis of the context inwhich the data is predicted. In still other embodiments, prediction andlearning may be hybrids, used in conjunction for providing contextuallyrelevant supplemental content to a vehicle, user account, user device,or some target associated with a user account or profile.

Overtime, machine learning can be used to reinforce learned behavior,which can provide weighting to certain inputs. For instance, the moretimes a user turns on the windshield wipers when it is raining, andwithin two minutes of turning on the car, may signal that this patternsis likely to happen again. In another example, if a user stops to chargehis vehicle at a particular charge station, which is 20 miles from hishome, repeatedly on Tuesdays, at 6 pm, when nobody is a passenger in thevehicle, and the vehicle had less than 5% charge, may be used as astrong pattern that this may occur again in the future. This data,combined with other data, may be used to recommend data regarding thecharge station in advance, so that the user need no look up the chargestation to reserve a spot, or the like. It should be understood thatthese are just simplified examples to convey examples of recommendationswhich may be based on some learning, preferences or pattern analysis, orlikelihoods.

Thus, context awareness across multiple dimensions will allow for moreaccurate predictions, learning (e.g., by building and refining behaviormodels), and surfacing/suggesting recommendations of supplementalcontent or settings, when it is most probable or likely or useful, orneeded by the user or vehicle and user, or relevant at a current orproximate or near or destination geo-location.

For purposes of providing example ways of processing learningalgorithms, machine learning methods, predictions, data analysis, andthe like, without limitations to any specifically claimed embodiment,reference may be made to a book entitled “Introduction to MachineLearning”, Second Edition, by Ethem Alpaydin, The MIT Press (ISBN978-0-262-01243-0), Cambridge, Massachusetts, London England (2010),which is herein incorporated by reference for all purposes. In oneembodiment, various methods for detecting emotion may use some of thetechnical computations described in one or more of the following papers,which are incorporated herein by reference. One example is described ina paper entitled “Emotion Recognition from Facial Expressions usingMultilevel HMM” by Ira Cohen, et al., published by The University ofIllinois at Urbana-Champaign, published in the year 2000. Anotherexample is described in a paper entitled “Human Emotion RecognitionSystem,” by Dilbag Singh, from the Computer Science and EngineeringDept. Guru Nanak Dev University Amritsar (Punjab) India, published in I.J. Image, Graphics and Signaling Processing, 2012, 8, 50-56. Anotherexample is described in a paper entitled “Recognizing emotion in speechusing neural networks”, by Keshi Dai, et al., from College of Computerand Information Science, Northeastern University, Boston, MA, publishedby Telehealth/AT '08 Proceedings of the IASTED International Conferenceon Telehealth/Assistive Technologies, 2008, Pages 31-36. Another exampleis described in a paper entitled “Speech Emotion Recognition Using DeepNeural Network and Extreme Learning Machine” by Kun Han et al.,Department of Computer Science and Engineering, The Ohio StateUniversity, Columbus, Ohio and Microsoft Research, published byInterspeech 2014. Yet another example is described in a paper entitled“Robust Recognition of Emotion from Speech,” by Mohammed E. Hogue, fromthe Department of Electrical and Computer Engineering, The University ofMemphis, published on 2006. In one embodiment, emotion may be determinedby also taking into account gender. Another example includes a paperentitled “Detecting Emotions in Conversations Between Driver and In-CarInformation Systems,” by Christian M. Jones, et al. School ofMathematical and Computer Sciences, Heriot-Watt University, Edinburgh,UK, and Department of Communications, Sanford University, CA, 2005. Oneother example is a paper describing detection of emotion is entitled“Emotion and Gender Recognition of Speech Signals Using SVM” by S.Sravan Kumar et al., published in the International Journal ofEngineering Science and Innovative Technology, Vol. 4, Issue 3, May2015. All of the papers and articles identified in this disclosure areincorporated by reference for all purposes.

In one embodiment, a display of a vehicle can include one or moredisplays. For example, a display screen of the vehicle may include anyone or more of a main dashboard display, or a center console display, ora combined main dashboard and center console display, or a surfacedisplay, or a glass surface, or a windshield display, or a windowdisplay, or a touch surface display, or a headrest display, or a movabledisplay, or a wireless display, or a wire-connected display, orcombinations thereof.

In one embodiment, biometrics may be associated to the user account. Thebiometrics may be used to monitor use of the vehicle and determine ifthe custom user interfaces is to be enabled, or if a guest custom userinterface is to be enabled, or if public custom user interface is to beenabled, or identify an interaction mode. The user account may includeprofile data defining when particular custom user interfaces are to beenabled or interactions modes are to be used. The biometrics may includeone or more of image data of a driver's face, a passenger's face, afinger print, a retina scan, a signature, a gesture, a user input, alogin, a key, a paring device, or combinations of two or more thereof.

The various embodiments defined herein may define individualimplementations or can define implementations that rely on combinationsof one or more of the defined embodiments. Further, embodiments of thepresent invention may be practiced with various computer systemconfigurations including hand-held devices, microprocessor systems,microprocessor-based or programmable consumer electronics,minicomputers, mainframe computers and the like. The invention can alsobe practiced in distributed computing environments where tasks areperformed by remote processing devices that are linked through awire-based or wireless network.

With the above embodiments in mind, it should be understood that theinvention could employ various computer-implemented operations involvingdata stored in computer systems. These operations are those requiringphysical manipulation of physical quantities. Usually, though notnecessarily, these quantities take the form of electrical or magneticsignals capable of being stored, transferred, combined, compared andotherwise manipulated.

Any of the operations described herein that form part of the inventionare useful machine operations. The invention also relates to a device oran apparatus for performing these operations. The apparatus can bespecially constructed for the required purpose, or the apparatus can bea general-purpose computer selectively activated or configured by acomputer program stored in the computer. In particular, variousgeneral-purpose machines can be used with computer programs written inaccordance with the teachings herein, or it may be more convenient toconstruct a more specialized apparatus to perform the requiredoperations.

The invention can also be embodied as computer readable code on acomputer readable medium. The computer readable medium is any datastorage device that can store data, which can thereafter be read by acomputer system. The computer readable medium can also be distributedover a network-coupled computer system so that the computer readablecode is stored and executed in a distributed fashion.

Although the foregoing invention has been described in some detail forpurposes of clarity of understanding, it will be apparent that certainchanges and modifications can be practiced within the scope of theappended claims. Accordingly, the present embodiments are to beconsidered as illustrative and not restrictive, and the invention is notto be limited to the details given herein, but may be modified withinthe scope and equivalents of the description and claims.

What is claimed is:
 1. A method for using an emotion of a human driverof a vehicle, comprising, processing captured voice data from the humandriver over a period of time, the processing of captured voice dataincludes generation of a user data graph for the human driver, the userdata graph includes learned voice profiles for the human driver;analyzing the voice data to identify a learned voice profile for thehuman driver, the learned voice profile assists in prediction of theemotion of the human driver; and generating instructions for processingcontrol of a vehicle response, the vehicle response selected in partbased on the emotion that was predicted for the human driver.
 2. Themethod of claim 1, wherein the processing of the captured voice data isperformed by a processor of a cloud services system.
 3. The method ofclaim 1, wherein the processing of the captured voice data is performedby a processor associated with the vehicle.
 4. The method of claim 1,wherein the user data graph is a model that takes into considerationcontextual information related to the human driver having interactionwith the vehicle and audible characteristics of voice of the human. 5.The method of claim 1, wherein the learned voice profile is customizedfor the human driver.
 6. The method of claim 1, wherein user data graphincorporates learning from a plurality of inputs, the plurality ofinputs include one or more of input patterns made for the vehicle,conversation input detected, language tone, language dialect, drivinghistory, biometrics, calendar data, vehicle preferences, learned vehiclepreferences, past mood patterns, and geo-location of the vehicle.
 7. Themethod of claim 1, further comprising, capturing ambient sounds in thevehicle, the ambient sounds include a voice of at least one passenger;and wherein the ambient sounds are processed in addition to the capturedvoice data from the human driver to assist in the determination of theemotion of the human driver.
 8. The method of claim 1, wherein thevehicle response is selected to assist in calming or reducingdistraction of the human driver.
 9. The method of claim 1, furthercomprising, processing the vehicle response includes causing an ambientlight change inside of the vehicle.
 10. The method of claim 1, whereinthe user data graph is a model that identifies characteristics of thehuman driver, the model uses as additional input one or more of;physiological conditions of the human driver that include, one or moreof heart rate, or body heat, or blood pressure, or skin temperature, orskin galvanic resistance, or temperature of skin, or blood rush to aface of the human driver, or eye iris or retina changes, or odor/scentchanges, or combinations of two or more thereof, or physical conditionsof the human driver that include, one or more of a change in a face ofthe human driver, or changes in a typing rhythm, or changes in gait, orchanges in voice, or changes in voice tone, or voice inflections, orvoice speed or slowness, or sharp voice harmonics, or voice identifiersor patterns, or elevated heat patterns, or palm veins, or eye iris orretina changes, heat rate changes, or eye gaze patterns or motions, orcombinations of two or more thereof.
 11. The method of claim 1, whereinthe vehicle response includes one of; waking up the human driver with asound or air if the emotion is a sleepy emotion, or generating warningsand/or signals to alert the human driver, or generating recommendationsto calm the human driver from an angry emotion or agitated emotion, orsending a notification to a third party indicating that the human driveris tired, or generating a notification to the human driver suggesting atemperature change, or generating an automatic temperature change basedon the emotion, or recommending the human driver stop driving whentired, or reducing recommendations to the human driver when thedetermined emotion is a rushed emotion, or changing lighting of thevehicle automatically, or changing temperature of the vehicleautomatically, or turn up or down a volume of music, or adjust a seatposition, or a combination of two or more thereof, and wherein at leastone of said vehicle responses are predefined to reduce distracteddriving or increase alertness of said human driver.
 12. The method ofclaim 1, wherein, the vehicle response is further based on current useof the vehicle, and use of the vehicle includes one or more ofgeolocation of the vehicle, speed of the vehicle, direction of thevehicle, a route of the vehicle, a time of day, occupants in thevehicle, historical use of the vehicle, online data of a user accessibleto the vehicle or server, or combinations of two or more thereof. 13.The method of claim 1, wherein a user account with a profile is providedfor the human driver of the vehicle, wherein the user account of thehuman driver is identified based on processing of credential dataobtained from the human driver, the credential data used by electronicsof the vehicle are to associated the human driver to the user accountand to the profile, the electronics of the vehicle configured tointerface over a network with a server for communication with theprofile and associate updates to the profile as made via use of thevehicle or updates associated with learned behavior of the human driver.14. The method of claim 1, wherein the analysis of the voice dataincludes examining the tone of the captured voice data to match the toneto one of said learned voice profiles of the human driver, the learnedvoice profiles include a voice tone of at least one of a rushed tone, afrustrated tone, a happy tone, a slow tone, a sleepy tone, a mad tone,an urgent tone, a normal tone.
 15. The method of claim 14, wherein thevehicle response is customized to be responsive to the voice tone. 16.The method of claim 14, wherein the voice tone is processed using noisefiltering and audio sampling, the processed voice tone produces an audiosignature that is used to identify the learned voice profile for thehuman driver.
 17. The method of claim 1, further comprising, examiningtouch input received on an input surface of the vehicle, the touch inputhaving a characteristic that is used to influence identification of thevehicle response.
 18. The method of claim 1, wherein the vehicleresponse is moderated in part by identification of one or moreconversation key words that are captured in the vehicle by the humandriver or the human driver and a passenger of the vehicle.
 19. Themethod of claim 1, wherein the vehicle response is moderated in part bypassive ambient conversation input captured in the vehicle.
 20. Themethod of claim 1, wherein the vehicle response includes interfacingwith one or more vehicle system functions, the interfacing includes oneor more of adjusting a setting of the vehicle, providing data to adisplay, recommending an input for a vehicle setting, adjusting an audiovolume, adjusting auto driver parameters, controlling speed settings, ora combination of two or more thereof.